[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:16:15 dist-test-slave-vcv1 krb5kdc[16640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:16:15 dist-test-slave-vcv1 krb5kdc[16640](info): set up 2 sockets
Sep 26 23:16:15 dist-test-slave-vcv1 krb5kdc[16640](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 23:16:16.469481 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:16:16 dist-test-slave-vcv1 krb5kdc[16640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:16:16.641326 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:46299
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:46299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:17.032541 16653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:17.033110 16653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:17.033383 16653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:17.044059 16653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:17.044365 16653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:17.044536 16653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:17.044721 16653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:17.044821 16653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:17.055702 16653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:46299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:46299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:17.057678 16653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:17.059413 16653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:17.077203 16658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:17.077102 16659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:17.077535 16661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:17.078538 16653 server_base.cc:1034] running on GCE node
I20260926 23:16:17.079530 16653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:17.081625 16653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:17.082882 16653 hybrid_clock.cc:648] HybridClock initialized: now 1790464577082838 us; error 59 us; skew 500 ppm
I20260926 23:16:17.083379 16653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:17 dist-test-slave-vcv1 krb5kdc[16640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:17.091568 16653 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:16:17.093544 16653 webserver.cc:460] Webserver started at http://127.16.62.126:41881/ using document root <none> and password file <none>
I20260926 23:16:17.094571 16653 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:17.094672 16653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:17.095229 16653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:17.099172 16653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "47a9ac502c2348f09173e3ab9a821840"
format_stamp: "Formatted at 2026-09-26 23:16:17 on dist-test-slave-vcv1"
server_key: "a80391e5d110cba9b66e517f063a7a56"
server_key_iv: "f21de7e90fa5e5b500149a6422cca419"
server_key_version: "encryptionkey@0"
I20260926 23:16:17.099926 16653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "47a9ac502c2348f09173e3ab9a821840"
format_stamp: "Formatted at 2026-09-26 23:16:17 on dist-test-slave-vcv1"
server_key: "a80391e5d110cba9b66e517f063a7a56"
server_key_iv: "f21de7e90fa5e5b500149a6422cca419"
server_key_version: "encryptionkey@0"
I20260926 23:16:17.107062 16653 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260926 23:16:17.112057 16668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:17.113850 16653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:16:17.114029 16653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "47a9ac502c2348f09173e3ab9a821840"
format_stamp: "Formatted at 2026-09-26 23:16:17 on dist-test-slave-vcv1"
server_key: "a80391e5d110cba9b66e517f063a7a56"
server_key_iv: "f21de7e90fa5e5b500149a6422cca419"
server_key_version: "encryptionkey@0"
I20260926 23:16:17.114233 16653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:17.136492 16653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:17.141271 16653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:17.141501 16653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:17.169955 16653 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:46299
I20260926 23:16:17.169975 16719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:46299 every 8 connection(s)
I20260926 23:16:17.171941 16653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:17.177886 16720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:17.178947 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 16653
I20260926 23:16:17.179279 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:17.181319 16633 external_mini_cluster.cc:1428] Setting key 8229bbcffb3ae1839c447b552c10507c
I20260926 23:16:17.192399 16720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840: Bootstrap starting.
I20260926 23:16:17.196785 16720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:17.198386 16720 log.cc:824] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:17.203912 16720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840: No bootstrap required, opened a new log
I20260926 23:16:17.215121 16720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a9ac502c2348f09173e3ab9a821840" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46299 } }
I20260926 23:16:17.215628 16720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 26 23:16:17 dist-test-slave-vcv1 krb5kdc[16640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:17.215966 16720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a9ac502c2348f09173e3ab9a821840, State: Initialized, Role: FOLLOWER
I20260926 23:16:17.217298 16720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a9ac502c2348f09173e3ab9a821840" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46299 } }
I20260926 23:16:17.217645 16720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:17.217944 16720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:17.218194 16720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:17.226642 16722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:17.186373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.16.62.126:46299)
0926 23:16:17.187464 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:17.187487 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:17.189143 (+ 1656us) server_negotiation.cc:407] Connection header received
0926 23:16:17.199420 (+ 10277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:17.199441 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:17.199983 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:17.200539 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:17.205008 (+ 4469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:17.206783 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:17.209315 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:17.210086 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:17.210273 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:17.217326 (+ 7053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:17.217397 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:17.217464 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:17.217530 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:17.221783 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:17.222702 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:17.222708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:17.222713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:17.222907 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:17.223361 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:17.223364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:17.223366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:17.223940 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:17.224337 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:17.225114 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 23:16:17.225406 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":394,"threads_started":1}
I20260926 23:16:17.230283 16720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a9ac502c2348f09173e3ab9a821840" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46299 } }
I20260926 23:16:17.231372 16720 leader_election.cc:302] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47a9ac502c2348f09173e3ab9a821840; no voters:
I20260926 23:16:17.232182 16720 leader_election.cc:288] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:17.232466 16725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:17.233055 16725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [term 1 LEADER]: Becoming Leader. State: Replica: 47a9ac502c2348f09173e3ab9a821840, State: Running, Role: LEADER
I20260926 23:16:17.233843 16725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a9ac502c2348f09173e3ab9a821840" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46299 } }
I20260926 23:16:17.235239 16720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:17.238427 16726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a9ac502c2348f09173e3ab9a821840. Latest consensus state: current_term: 1 leader_uuid: "47a9ac502c2348f09173e3ab9a821840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a9ac502c2348f09173e3ab9a821840" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46299 } } }
I20260926 23:16:17.238958 16726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:17.240099 16727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47a9ac502c2348f09173e3ab9a821840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a9ac502c2348f09173e3ab9a821840" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46299 } } }
I20260926 23:16:17.240474 16727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:17.242069 16733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:17.249624 16733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:17.261879 16733 catalog_manager.cc:1347] Generated new cluster ID: ecdb9f44f3ef4a32abbf9e77e8620578
I20260926 23:16:17.262007 16733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:17.284498 16733 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:17.285975 16733 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:17.299664 16733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47a9ac502c2348f09173e3ab9a821840: Generated new TSK 0
I20260926 23:16:17.300701 16733 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
W20260926 23:16:18.490726 16633 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.62.65: real 1.181s user 0.001s sys 0.010s
I20260926 23:16:18.491863 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:46299
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:18.855818 16746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:18.856125 16746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:18.856446 16746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:18.863760 16746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:18.863862 16746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:18.864082 16746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:16:18.873876 16746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:46299
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:18.875187 16746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:18.876540 16746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:18.890424 16751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:18.890373 16752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:18.890401 16754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:18.891160 16746 server_base.cc:1034] running on GCE node
I20260926 23:16:18.891805 16746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:18.892802 16746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:18.894027 16746 hybrid_clock.cc:648] HybridClock initialized: now 1790464578893993 us; error 46 us; skew 500 ppm
I20260926 23:16:18.894344 16746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:18 dist-test-slave-vcv1 krb5kdc[16640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464578, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:18.897452 16746 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:16:18.898862 16746 webserver.cc:460] Webserver started at http://127.16.62.65:37721/ using document root <none> and password file <none>
I20260926 23:16:18.899537 16746 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:18.899683 16746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:18.900002 16746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:18.902906 16746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "23de5e6617a044e48c1fd5ec5ef1aef3"
format_stamp: "Formatted at 2026-09-26 23:16:18 on dist-test-slave-vcv1"
server_key: "82d2317b8f34657c28970f7213f671d0"
server_key_iv: "c97d9a2249c4f22b172b97b286dca799"
server_key_version: "encryptionkey@0"
I20260926 23:16:18.903765 16746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "23de5e6617a044e48c1fd5ec5ef1aef3"
format_stamp: "Formatted at 2026-09-26 23:16:18 on dist-test-slave-vcv1"
server_key: "82d2317b8f34657c28970f7213f671d0"
server_key_iv: "c97d9a2249c4f22b172b97b286dca799"
server_key_version: "encryptionkey@0"
I20260926 23:16:18.908731 16746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 23:16:18.911970 16761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:18.913239 16746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:18.913403 16746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "23de5e6617a044e48c1fd5ec5ef1aef3"
format_stamp: "Formatted at 2026-09-26 23:16:18 on dist-test-slave-vcv1"
server_key: "82d2317b8f34657c28970f7213f671d0"
server_key_iv: "c97d9a2249c4f22b172b97b286dca799"
server_key_version: "encryptionkey@0"
I20260926 23:16:18.913597 16746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:18.930451 16746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:18.934108 16746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:18.934384 16746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:18.935400 16746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:18.936947 16746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:18.937048 16746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:18.937273 16746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:18.937353 16746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:18.993320 16746 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:41737
I20260926 23:16:18.993348 16873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:41737 every 8 connection(s)
I20260926 23:16:18.994751 16746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:16:18.998986 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 16746
I20260926 23:16:18.999185 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:16:19.000007 16633 external_mini_cluster.cc:1428] Setting key a8f81b51a51e4f5602bd255839dc5bfa
Sep 26 23:16:19 dist-test-slave-vcv1 krb5kdc[16640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464578, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
I20260926 23:16:19.019343 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:18.997743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:41213 (local address 127.16.62.126:46299)
0926 23:16:18.998179 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:18.998184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:18.999399 (+ 1215us) server_negotiation.cc:407] Connection header received
0926 23:16:19.001383 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:19.001388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:19.001486 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:19.001624 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:19.003102 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:19.003813 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:19.005379 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:19.005891 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:19.006028 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:19.009826 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:19.009835 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:19.009843 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:19.009887 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:19.015290 (+ 5403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:19.016039 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:19.016046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:19.016051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:19.016377 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:19.016875 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:19.016880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:19.016884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:19.017413 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:19.017626 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:19.018497 (+ 871us) server_negotiation.cc:299] Negotiation successful
0926 23:16:19.018739 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":216,"threads_started":1}
I20260926 23:16:19.020306 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:18.998291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:46299 (local address 127.16.62.65:41213)
0926 23:16:18.999038 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:18.999271 (+ 233us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:19.001128 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:19.001862 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:19.001880 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:19.002374 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:19.002926 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:19.002941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:19.003973 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:19.003979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:19.005237 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:19.005248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:19.006145 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:19.006166 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:19.006446 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:19.007149 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:19.007179 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:19.009518 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:19.015468 (+ 5950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:19.015475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:19.015486 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:19.015897 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:19.016540 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:19.016543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:19.016546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:19.016746 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:19.017579 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:19.017587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:19.017994 (+ 407us) client_negotiation.cc:769] Sending connection context
0926 23:16:19.018358 (+ 364us) client_negotiation.cc:240] Negotiation successful
0926 23:16:19.018826 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":320,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
I20260926 23:16:19.022217 16874 heartbeater.cc:347] Connected to a master server at 127.16.62.126:46299
I20260926 23:16:19.022671 16874 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:19.023759 16874 heartbeater.cc:511] Master 127.16.62.126:46299 requested a full tablet report, sending...
I20260926 23:16:19.027417 16685 ts_manager.cc:194] Registered new tserver with Master: 23de5e6617a044e48c1fd5ec5ef1aef3 (127.16.62.65:41737)
I20260926 23:16:19.029631 16685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:41213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:16:19.043525 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:46299
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:19.408003 16880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:19.408356 16880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:19.408519 16880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:19.415648 16880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:19.415829 16880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:19.416028 16880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:16:19.425869 16880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:46299
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:19.427066 16880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:19.428496 16880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:19.442011 16885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:19.442039 16888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:19.442044 16886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:19.442891 16880 server_base.cc:1034] running on GCE node
I20260926 23:16:19.443431 16880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:19.444330 16880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:19.445546 16880 hybrid_clock.cc:648] HybridClock initialized: now 1790464579445503 us; error 48 us; skew 500 ppm
I20260926 23:16:19.445868 16880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:19 dist-test-slave-vcv1 krb5kdc[16640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464579, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:19.449295 16880 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:16:19.450743 16880 webserver.cc:460] Webserver started at http://127.16.62.66:37203/ using document root <none> and password file <none>
I20260926 23:16:19.451431 16880 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:19.451574 16880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:19.451875 16880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:19.454720 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "3039b0240e044a1ebca3475b0ba30ea4"
format_stamp: "Formatted at 2026-09-26 23:16:19 on dist-test-slave-vcv1"
server_key: "86fd7b102dea6cd355b19a2c2f80ada3"
server_key_iv: "9816905a29095063b4d0b3162e5664e0"
server_key_version: "encryptionkey@0"
I20260926 23:16:19.455444 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "3039b0240e044a1ebca3475b0ba30ea4"
format_stamp: "Formatted at 2026-09-26 23:16:19 on dist-test-slave-vcv1"
server_key: "86fd7b102dea6cd355b19a2c2f80ada3"
server_key_iv: "9816905a29095063b4d0b3162e5664e0"
server_key_version: "encryptionkey@0"
I20260926 23:16:19.460438 16880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 23:16:19.463518 16895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:19.464741 16880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 23:16:19.464905 16880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "3039b0240e044a1ebca3475b0ba30ea4"
format_stamp: "Formatted at 2026-09-26 23:16:19 on dist-test-slave-vcv1"
server_key: "86fd7b102dea6cd355b19a2c2f80ada3"
server_key_iv: "9816905a29095063b4d0b3162e5664e0"
server_key_version: "encryptionkey@0"
I20260926 23:16:19.465096 16880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:19.480237 16880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:19.483645 16880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:19.483973 16880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:19.484856 16880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:19.486050 16880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:19.486150 16880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:19.486299 16880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:19.486367 16880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:19.535753 16880 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:38995
I20260926 23:16:19.535810 17007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:38995 every 8 connection(s)
I20260926 23:16:19.537144 16880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:16:19.540496 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 16880
I20260926 23:16:19.540657 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:16:19.541074 16633 external_mini_cluster.cc:1428] Setting key acd7513a07c046f97f9bb00605aa8789
Sep 26 23:16:19 dist-test-slave-vcv1 krb5kdc[16640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464579, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
I20260926 23:16:19.559160 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:19.539729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:57133 (local address 127.16.62.126:46299)
0926 23:16:19.540200 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:19.540206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:19.541398 (+ 1192us) server_negotiation.cc:407] Connection header received
0926 23:16:19.543401 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:19.543406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:19.543495 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:19.543623 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:19.545288 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:19.546395 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:19.549000 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:19.549423 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:19.549516 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:19.553630 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:19.553635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:19.553639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:19.553669 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:19.556199 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:19.557109 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:19.557112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:19.557115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:19.557227 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:19.557712 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:19.557715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:19.557717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:19.557955 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:19.558059 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:19.558750 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 23:16:19.558918 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260926 23:16:19.560128 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:19.540384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:46299 (local address 127.16.62.66:57133)
0926 23:16:19.541213 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:19.541269 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:19.543137 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:19.543819 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:19.543827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:19.544421 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:19.545100 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:19.545130 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:19.546652 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:19.546656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:19.548857 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:19.548875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:19.549638 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:19.549647 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:19.549983 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:19.550698 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:19.550739 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:19.553332 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:19.556338 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:19.556346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:19.556362 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:19.556992 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:19.557344 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:19.557350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:19.557353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:19.557596 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:19.558063 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:19.558068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:19.558371 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 23:16:19.558635 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 23:16:19.558994 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":370,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
I20260926 23:16:19.561616 17008 heartbeater.cc:347] Connected to a master server at 127.16.62.126:46299
I20260926 23:16:19.561941 17008 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:19.562862 17008 heartbeater.cc:511] Master 127.16.62.126:46299 requested a full tablet report, sending...
I20260926 23:16:19.564437 16685 ts_manager.cc:194] Registered new tserver with Master: 3039b0240e044a1ebca3475b0ba30ea4 (127.16.62.66:38995)
I20260926 23:16:19.565199 16685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:57133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:16:19.582882 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:46299
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:19.973905 17014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:19.974210 17014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:19.974382 17014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:19.981530 17014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:19.981666 17014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:19.981841 17014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:16:19.991092 17014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:46299
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:19.992283 17014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:19.993551 17014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:20.005852 17020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:20.005806 17019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:20.006176 17022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:20.006538 17014 server_base.cc:1034] running on GCE node
I20260926 23:16:20.007089 17014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:20.008044 17014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:20.009256 17014 hybrid_clock.cc:648] HybridClock initialized: now 1790464580009193 us; error 49 us; skew 500 ppm
I20260926 23:16:20.009505 17014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[16640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:20.012375 17014 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:16:20.013628 17014 webserver.cc:460] Webserver started at http://127.16.62.67:35665/ using document root <none> and password file <none>
I20260926 23:16:20.014178 17014 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:20.014277 17014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:20.014513 17014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:20.017498 17014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "67ad22463d05447c9f5f15e1af3a0857"
format_stamp: "Formatted at 2026-09-26 23:16:20 on dist-test-slave-vcv1"
server_key: "53f748423d9d56ec566a7b5a4736e3d3"
server_key_iv: "89417bf933ef91955b4ff83f23cbfbae"
server_key_version: "encryptionkey@0"
I20260926 23:16:20.018329 17014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "67ad22463d05447c9f5f15e1af3a0857"
format_stamp: "Formatted at 2026-09-26 23:16:20 on dist-test-slave-vcv1"
server_key: "53f748423d9d56ec566a7b5a4736e3d3"
server_key_iv: "89417bf933ef91955b4ff83f23cbfbae"
server_key_version: "encryptionkey@0"
I20260926 23:16:20.023254 17014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 23:16:20.026891 17029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:20.028136 17014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:16:20.028342 17014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "67ad22463d05447c9f5f15e1af3a0857"
format_stamp: "Formatted at 2026-09-26 23:16:20 on dist-test-slave-vcv1"
server_key: "53f748423d9d56ec566a7b5a4736e3d3"
server_key_iv: "89417bf933ef91955b4ff83f23cbfbae"
server_key_version: "encryptionkey@0"
I20260926 23:16:20.028539 17014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:20.033144 16874 heartbeater.cc:503] Master 127.16.62.126:46299 was elected leader, sending a full tablet report...
I20260926 23:16:20.057052 17014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:20.063287 17014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:20.063783 17014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:20.065173 17014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:20.066949 17014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:20.067122 17014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:20.067358 17014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:20.067493 17014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:20.112406 17014 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:46829
I20260926 23:16:20.112448 17141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:46829 every 8 connection(s)
I20260926 23:16:20.113644 17014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:16:20.119869 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 17014
I20260926 23:16:20.120047 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:16:20.120537 16633 external_mini_cluster.cc:1428] Setting key 79dd626817b77cc67c4051706d1cc9f9
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[16640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:20.133088 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:20.116172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:60159 (local address 127.16.62.126:46299)
0926 23:16:20.116576 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:20.116580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:20.117714 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 23:16:20.119724 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:20.119729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:20.119815 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:20.119932 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:20.121470 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.122409 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.124275 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.124645 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.124772 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:20.127822 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:20.127827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:20.127830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:20.127857 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:20.130240 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.130927 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.130939 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.130942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.131041 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.131436 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.131440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.131443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.131678 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:20.131832 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:20.132490 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 23:16:20.132685 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":184,"threads_started":1}
I20260926 23:16:20.133935 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:20.116717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:46299 (local address 127.16.62.67:60159)
0926 23:16:20.117560 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:20.117596 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:20.119416 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:20.120202 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:20.120218 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:20.120766 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:20.121311 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:20.121326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.122561 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.122572 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:20.124078 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:20.124087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.124858 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.124865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:20.125083 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:20.125661 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:20.125681 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:20.127618 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:20.130367 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:20.130375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:20.130390 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:20.130795 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:20.131149 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:20.131154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:20.131158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:20.131325 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:20.131817 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:20.131824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:20.132107 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 23:16:20.132407 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 23:16:20.132894 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":342,"threads_started":1}
I20260926 23:16:20.135354 17142 heartbeater.cc:347] Connected to a master server at 127.16.62.126:46299
I20260926 23:16:20.135663 17142 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:20.136518 17142 heartbeater.cc:511] Master 127.16.62.126:46299 requested a full tablet report, sending...
I20260926 23:16:20.137758 16685 ts_manager.cc:194] Registered new tserver with Master: 67ad22463d05447c9f5f15e1af3a0857 (127.16.62.67:46829)
I20260926 23:16:20.138532 16685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:60159
I20260926 23:16:20.148461 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[16640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464580, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[16640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464580, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:20.190117 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:20.176026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.16.62.65:41737)
0926 23:16:20.176948 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:20.176957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:20.176988 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:16:20.177057 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:20.177066 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:20.177346 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:20.177512 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:20.178791 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.179768 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.181042 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.181589 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.181695 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:20.185339 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:20.185349 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:20.185363 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:20.185391 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:20.187884 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.188510 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.188516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.188518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.188626 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.188952 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.188955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.188957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.189254 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:20.189471 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:20.189680 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 23:16:20.189851 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":600,"threads_started":1}
W20260926 23:16:20.191329 16789 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48252
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[16640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[16640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:20.232975 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:20.219274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.16.62.65:41737)
0926 23:16:20.219482 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:20.219489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:20.219831 (+ 342us) server_negotiation.cc:407] Connection header received
0926 23:16:20.220051 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:20.220056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:20.220121 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:20.220283 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:20.221558 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.222238 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.223429 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.223841 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.223980 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:20.227784 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:20.227789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:20.227792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:20.227825 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:20.230765 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.231342 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.231345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.231348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.231462 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.231765 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.231769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.231772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.232045 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:20.232291 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:20.232536 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 23:16:20.232686 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 23:16:20.234561 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 16746
I20260926 23:16:20.246284 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:20.235397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.16.62.65:41737)
0926 23:16:20.235535 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:20.235538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:20.235917 (+ 379us) server_negotiation.cc:407] Connection header received
0926 23:16:20.236024 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:20.236027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:20.236076 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:20.236202 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:20.237415 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.238090 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.239315 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.239733 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.239826 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:20.241993 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:20.241997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:20.242000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:20.242027 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:20.244332 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.244861 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.244864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.244867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.244983 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.245266 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.245268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.245270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.245482 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:20.245643 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:20.245857 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 23:16:20.245994 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 23:16:20.247071 16869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:20.332408 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 16746
I20260926 23:16:20.349254 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 16880
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[16640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:16:20.364399 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:20.350102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.16.62.66:38995)
0926 23:16:20.350558 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:20.350565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:20.350587 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:16:20.350644 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:20.350650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:20.350889 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:20.351052 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:20.352211 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.353146 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.354370 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.354899 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.355044 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:20.359005 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:20.359015 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:20.359028 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:20.359058 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:20.361636 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.362300 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.362315 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.362319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.362446 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.362824 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.362831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.362834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.363144 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:20.363438 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:20.363759 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 23:16:20.364025 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":232,"threads_started":1}
I20260926 23:16:20.365587 17003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:20.445770 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 16880
I20260926 23:16:20.463608 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17014
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[16640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:16:20.479307 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:20.464804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.16.62.67:46829)
0926 23:16:20.465423 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:20.465429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:20.465458 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:16:20.465511 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:20.465518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:20.465718 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:20.465838 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:20.467128 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.468067 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.469477 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:20.470003 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:20.470116 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:20.474386 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:20.474396 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:20.474410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:20.474440 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:20.477189 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.477728 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.477741 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.477743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.477845 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:20.478135 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:20.478138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:20.478139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:20.478441 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:20.478661 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:20.478855 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 23:16:20.479033 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":327,"threads_started":1}
I20260926 23:16:20.480548 17137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:20.561847 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 17014
I20260926 23:16:20.579385 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 16653
I20260926 23:16:20.580580 16715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:20.680953 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 16653
2026-09-26T23:16:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5303 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[17164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[17164](info): set up 2 sockets
Sep 26 23:16:20 dist-test-slave-vcv1 krb5kdc[17164](info): commencing operation
krb5kdc: starting...
W20260926 23:16:21.776988 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:16:21 dist-test-slave-vcv1 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:16:21.951650 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:40383
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:40383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:22.289897 17177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:22.290179 17177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:22.290264 17177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:22.297254 17177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:22.297363 17177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:22.297407 17177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:22.297456 17177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:22.297556 17177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:22.307163 17177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:40383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:22.308485 17177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:22.309760 17177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:22.321169 17183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:22.321201 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:22.321875 17185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:22.322136 17177 server_base.cc:1034] running on GCE node
I20260926 23:16:22.322906 17177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:22.324232 17177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:22.325485 17177 hybrid_clock.cc:648] HybridClock initialized: now 1790464582325453 us; error 44 us; skew 500 ppm
I20260926 23:16:22.325839 17177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:22 dist-test-slave-vcv1 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:22.336848 17177 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:16:22.338461 17177 webserver.cc:460] Webserver started at http://127.16.62.126:42075/ using document root <none> and password file <none>
I20260926 23:16:22.339175 17177 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:22.339325 17177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:22.339650 17177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:22.342774 17177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "f6f838c3eb294013903ea8ba213303cb"
format_stamp: "Formatted at 2026-09-26 23:16:22 on dist-test-slave-vcv1"
server_key: "b59d41cb5c776d3d2b998a2f21946da1"
server_key_iv: "97b526a246455312552644309725e002"
server_key_version: "encryptionkey@0"
I20260926 23:16:22.343518 17177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "f6f838c3eb294013903ea8ba213303cb"
format_stamp: "Formatted at 2026-09-26 23:16:22 on dist-test-slave-vcv1"
server_key: "b59d41cb5c776d3d2b998a2f21946da1"
server_key_iv: "97b526a246455312552644309725e002"
server_key_version: "encryptionkey@0"
I20260926 23:16:22.348609 17177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 23:16:22.352041 17192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:22.353608 17177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:22.353790 17177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "f6f838c3eb294013903ea8ba213303cb"
format_stamp: "Formatted at 2026-09-26 23:16:22 on dist-test-slave-vcv1"
server_key: "b59d41cb5c776d3d2b998a2f21946da1"
server_key_iv: "97b526a246455312552644309725e002"
server_key_version: "encryptionkey@0"
I20260926 23:16:22.354003 17177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:22.374794 17177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:22.378443 17177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:22.378714 17177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:22.396373 17177 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:40383
I20260926 23:16:22.396402 17243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:40383 every 8 connection(s)
I20260926 23:16:22.397680 17177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:22.399601 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 17177
I20260926 23:16:22.399816 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:22.400660 16633 external_mini_cluster.cc:1428] Setting key 9fb76be1765d471701b3a0050bbe478b
I20260926 23:16:22.402084 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:22.413103 17244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb: Bootstrap starting.
Sep 26 23:16:22 dist-test-slave-vcv1 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:22.416778 17244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:22.417821 17244 log.cc:824] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:22.420850 17244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb: No bootstrap required, opened a new log
I20260926 23:16:22.422009 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:22.402680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.16.62.126:40383)
0926 23:16:22.403370 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:22.403380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:22.403420 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:16:22.404976 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:22.404995 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:22.405399 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:22.405827 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:22.407250 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:22.408628 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:22.409813 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:22.410474 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:22.410685 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:22.414965 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:22.414976 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:22.414989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:22.415029 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:22.418193 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:22.418880 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:22.418890 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:22.418900 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:22.419076 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:22.419465 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:22.419468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:22.419470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:22.419928 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:22.420191 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:22.420730 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 23:16:22.421074 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":313,"threads_started":1}
I20260926 23:16:22.428011 17244 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f838c3eb294013903ea8ba213303cb" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40383 } }
I20260926 23:16:22.428418 17244 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:22.428531 17244 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f838c3eb294013903ea8ba213303cb, State: Initialized, Role: FOLLOWER
I20260926 23:16:22.429055 17244 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f838c3eb294013903ea8ba213303cb" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40383 } }
I20260926 23:16:22.429275 17244 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:22.429405 17244 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:22.429575 17244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:22.431653 17244 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f838c3eb294013903ea8ba213303cb" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40383 } }
I20260926 23:16:22.432130 17244 leader_election.cc:302] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f838c3eb294013903ea8ba213303cb; no voters:
I20260926 23:16:22.432847 17244 leader_election.cc:288] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:22.432933 17249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:22.433538 17249 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [term 1 LEADER]: Becoming Leader. State: Replica: f6f838c3eb294013903ea8ba213303cb, State: Running, Role: LEADER
I20260926 23:16:22.434113 17249 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f838c3eb294013903ea8ba213303cb" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40383 } }
I20260926 23:16:22.435035 17244 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:22.437018 17251 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f838c3eb294013903ea8ba213303cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f838c3eb294013903ea8ba213303cb" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40383 } } }
I20260926 23:16:22.437397 17251 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:22.437124 17250 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f838c3eb294013903ea8ba213303cb. Latest consensus state: current_term: 1 leader_uuid: "f6f838c3eb294013903ea8ba213303cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f838c3eb294013903ea8ba213303cb" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40383 } } }
I20260926 23:16:22.437531 17250 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:22.441104 17254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:22.446782 17254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:22.455199 17254 catalog_manager.cc:1347] Generated new cluster ID: 4f132deac0904ffbb98461db1e48825e
I20260926 23:16:22.455334 17254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:22.482625 17254 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:22.484532 17254 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:22.495412 17254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f838c3eb294013903ea8ba213303cb: Generated new TSK 0
I20260926 23:16:22.496451 17254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:16:22.545753 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40383
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:22.861196 17270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:22.861570 17270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:22.861748 17270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:22.869004 17270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:22.869179 17270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:22.869364 17270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:16:22.878855 17270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40383
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:22.880280 17270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:22.881610 17270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:22.894322 17275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:22.894299 17276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:22.894277 17278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:22.895323 17270 server_base.cc:1034] running on GCE node
I20260926 23:16:22.895874 17270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:22.896797 17270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:22.898021 17270 hybrid_clock.cc:648] HybridClock initialized: now 1790464582897986 us; error 50 us; skew 500 ppm
I20260926 23:16:22.898372 17270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:22 dist-test-slave-vcv1 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:22.901420 17270 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:16:22.902735 17270 webserver.cc:460] Webserver started at http://127.16.62.65:42601/ using document root <none> and password file <none>
I20260926 23:16:22.903399 17270 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:22.903537 17270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:22.903844 17270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:22.906625 17270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "e6f0f8c211064236ae97e775939dbc85"
format_stamp: "Formatted at 2026-09-26 23:16:22 on dist-test-slave-vcv1"
server_key: "75325e370c60388ce3b402d5999475b8"
server_key_iv: "e8c0f4a85f57c16ab848ff055c19b030"
server_key_version: "encryptionkey@0"
I20260926 23:16:22.907406 17270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "e6f0f8c211064236ae97e775939dbc85"
format_stamp: "Formatted at 2026-09-26 23:16:22 on dist-test-slave-vcv1"
server_key: "75325e370c60388ce3b402d5999475b8"
server_key_iv: "e8c0f4a85f57c16ab848ff055c19b030"
server_key_version: "encryptionkey@0"
I20260926 23:16:22.912333 17270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:16:22.915463 17285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:22.916808 17270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:22.916973 17270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "e6f0f8c211064236ae97e775939dbc85"
format_stamp: "Formatted at 2026-09-26 23:16:22 on dist-test-slave-vcv1"
server_key: "75325e370c60388ce3b402d5999475b8"
server_key_iv: "e8c0f4a85f57c16ab848ff055c19b030"
server_key_version: "encryptionkey@0"
I20260926 23:16:22.917156 17270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:22.956322 17270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:22.960474 17270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:22.960784 17270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:22.961779 17270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:22.963081 17270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:22.963192 17270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:22.963353 17270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:22.963425 17270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:22.995926 17270 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:41971
I20260926 23:16:22.995980 17397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:41971 every 8 connection(s)
I20260926 23:16:22.997242 17270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:16:23.002099 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 17270
I20260926 23:16:23.002301 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:16:23.003060 16633 external_mini_cluster.cc:1428] Setting key 5f18741d264a12a6c99e28ffb3be5f92
Sep 26 23:16:23 dist-test-slave-vcv1 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:23.019640 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:22.999765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:59893 (local address 127.16.62.126:40383)
0926 23:16:23.000180 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:23.000187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:23.001205 (+ 1018us) server_negotiation.cc:407] Connection header received
0926 23:16:23.003689 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:23.003695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:23.003792 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:23.003907 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:23.005681 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.006890 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.008408 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.009134 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.009278 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:23.013722 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:23.013727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:23.013730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:23.013760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:23.016441 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:23.017390 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:23.017394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:23.017396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:23.017509 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:23.018008 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:23.018011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:23.018013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:23.018249 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:23.018364 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:23.019243 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 23:16:23.019391 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":180,"threads_started":1}
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
I20260926 23:16:23.021183 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:23.000415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:40383 (local address 127.16.62.65:59893)
0926 23:16:23.001059 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:23.001095 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:23.003381 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:23.004345 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:23.004353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:23.004954 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:23.005513 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:23.005530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.007021 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.007024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:23.008277 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:23.008289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.009377 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.009387 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:23.009754 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:23.010483 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:23.010515 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:23.013508 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:23.016571 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:23.016582 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:23.016601 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:23.017270 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:23.017616 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:23.017625 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:23.017631 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:23.017898 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:23.018386 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:23.018393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:23.018844 (+ 451us) client_negotiation.cc:769] Sending connection context
0926 23:16:23.019164 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 23:16:23.019681 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":265,"threads_started":1}
I20260926 23:16:23.023025 17398 heartbeater.cc:347] Connected to a master server at 127.16.62.126:40383
I20260926 23:16:23.023362 17398 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
I20260926 23:16:23.024344 17398 heartbeater.cc:511] Master 127.16.62.126:40383 requested a full tablet report, sending...
I20260926 23:16:23.026865 17209 ts_manager.cc:194] Registered new tserver with Master: e6f0f8c211064236ae97e775939dbc85 (127.16.62.65:41971)
I20260926 23:16:23.028970 17209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:59893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:16:23.046661 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40383
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:23.354547 17404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:23.354882 17404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:23.355067 17404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:23.362607 17404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:23.362742 17404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:23.362920 17404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:16:23.372462 17404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40383
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:23.373705 17404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:23.374989 17404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:23.387630 17412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:23.387655 17410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:23.388273 17409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:23.388515 17404 server_base.cc:1034] running on GCE node
I20260926 23:16:23.389163 17404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:23.390035 17404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:23.391247 17404 hybrid_clock.cc:648] HybridClock initialized: now 1790464583391200 us; error 57 us; skew 500 ppm
I20260926 23:16:23.391541 17404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:23 dist-test-slave-vcv1 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:23.394475 17404 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:16:23.395925 17404 webserver.cc:460] Webserver started at http://127.16.62.66:38523/ using document root <none> and password file <none>
I20260926 23:16:23.396638 17404 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:23.396782 17404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:23.397087 17404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:23.399744 17404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "371567e73d114132a4932c9ac1e7521b"
format_stamp: "Formatted at 2026-09-26 23:16:23 on dist-test-slave-vcv1"
server_key: "dd1904aae7433de09f5bc3f110439c72"
server_key_iv: "7ef25598f850284e7e55c1945f44b846"
server_key_version: "encryptionkey@0"
I20260926 23:16:23.400611 17404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "371567e73d114132a4932c9ac1e7521b"
format_stamp: "Formatted at 2026-09-26 23:16:23 on dist-test-slave-vcv1"
server_key: "dd1904aae7433de09f5bc3f110439c72"
server_key_iv: "7ef25598f850284e7e55c1945f44b846"
server_key_version: "encryptionkey@0"
I20260926 23:16:23.405431 17404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 23:16:23.408771 17419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:23.410041 17404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:16:23.410205 17404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "371567e73d114132a4932c9ac1e7521b"
format_stamp: "Formatted at 2026-09-26 23:16:23 on dist-test-slave-vcv1"
server_key: "dd1904aae7433de09f5bc3f110439c72"
server_key_iv: "7ef25598f850284e7e55c1945f44b846"
server_key_version: "encryptionkey@0"
I20260926 23:16:23.410406 17404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:23.425117 17404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:23.428686 17404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:23.428947 17404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:23.429811 17404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:23.431125 17404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:23.431234 17404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:23.431380 17404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:23.431460 17404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:23.464176 17404 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:39089
I20260926 23:16:23.464205 17531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:39089 every 8 connection(s)
I20260926 23:16:23.465425 17404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:16:23.471868 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 17404
I20260926 23:16:23.472043 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:16:23.472628 16633 external_mini_cluster.cc:1428] Setting key f7332e80cd6917cab571e9db3a69b658
Sep 26 23:16:23 dist-test-slave-vcv1 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:23.488339 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:23.467984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:51949 (local address 127.16.62.126:40383)
0926 23:16:23.468222 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:23.468237 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:23.469355 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 23:16:23.471041 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:23.471046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:23.471109 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:23.471237 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:23.473365 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.474430 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.476897 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.477326 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.477416 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:23.481504 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:23.481514 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:23.481521 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:23.481564 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:23.485102 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:23.485788 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:23.485796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:23.485802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:23.485953 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:23.486408 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:23.486416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:23.486423 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:23.486802 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:23.486974 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:23.487596 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 23:16:23.487864 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:16:23.488926 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:23.468519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:40383 (local address 127.16.62.66:51949)
0926 23:16:23.469196 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:23.469239 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:23.470808 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:23.471622 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:23.471636 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:23.472403 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:23.473179 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:23.473206 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.474585 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.474592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:23.476759 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:23.476777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.477540 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.477549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:23.477892 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:23.478599 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:23.478631 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:23.481273 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:23.485248 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:23.485255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:23.485267 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:23.485651 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:23.486059 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:23.486062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:23.486064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:23.486261 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:23.486917 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:23.486922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:23.487208 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 23:16:23.487441 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 23:16:23.487792 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":262,"threads_started":1}
I20260926 23:16:23.490377 17532 heartbeater.cc:347] Connected to a master server at 127.16.62.126:40383
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
I20260926 23:16:23.490700 17532 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:23.491397 17532 heartbeater.cc:511] Master 127.16.62.126:40383 requested a full tablet report, sending...
I20260926 23:16:23.492837 17209 ts_manager.cc:194] Registered new tserver with Master: 371567e73d114132a4932c9ac1e7521b (127.16.62.66:39089)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
I20260926 23:16:23.493686 17209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:51949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:16:23.515661 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40383
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:23.827627 17537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:23.827893 17537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:23.828050 17537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:23.835085 17537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:23.835212 17537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:23.835410 17537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:16:23.844805 17537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40383
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:23.846043 17537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:23.847339 17537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:23.860160 17543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:23.860297 17545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:23.860630 17542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:23.861377 17537 server_base.cc:1034] running on GCE node
I20260926 23:16:23.861984 17537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:23.862792 17537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:23.863978 17537 hybrid_clock.cc:648] HybridClock initialized: now 1790464583863937 us; error 32 us; skew 500 ppm
I20260926 23:16:23.864318 17537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:23 dist-test-slave-vcv1 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:23.867301 17537 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:16:23.868606 17537 webserver.cc:460] Webserver started at http://127.16.62.67:41503/ using document root <none> and password file <none>
I20260926 23:16:23.869213 17537 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:23.869307 17537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:23.869544 17537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:23.872364 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9"
format_stamp: "Formatted at 2026-09-26 23:16:23 on dist-test-slave-vcv1"
server_key: "1e2500e8090fc9b3404068c256db433c"
server_key_iv: "601f5b127c3049fe115ce2ebd2c9e86a"
server_key_version: "encryptionkey@0"
I20260926 23:16:23.873147 17537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9"
format_stamp: "Formatted at 2026-09-26 23:16:23 on dist-test-slave-vcv1"
server_key: "1e2500e8090fc9b3404068c256db433c"
server_key_iv: "601f5b127c3049fe115ce2ebd2c9e86a"
server_key_version: "encryptionkey@0"
I20260926 23:16:23.878029 17537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:16:23.881259 17552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:23.882450 17537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:16:23.882627 17537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9"
format_stamp: "Formatted at 2026-09-26 23:16:23 on dist-test-slave-vcv1"
server_key: "1e2500e8090fc9b3404068c256db433c"
server_key_iv: "601f5b127c3049fe115ce2ebd2c9e86a"
server_key_version: "encryptionkey@0"
I20260926 23:16:23.882817 17537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:23.903795 17537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:23.907836 17537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:23.908218 17537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:23.909128 17537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:23.910336 17537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:23.910404 17537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:23.910483 17537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:23.910534 17537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:23.942989 17537 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:43587
I20260926 23:16:23.943022 17664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:43587 every 8 connection(s)
I20260926 23:16:23.944307 17537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:16:23.951143 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 17537
I20260926 23:16:23.951318 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:16:23.951840 16633 external_mini_cluster.cc:1428] Setting key 340f2ac22325e3996a6a42e87cf16916
Sep 26 23:16:23 dist-test-slave-vcv1 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464583, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:23.963580 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:23.946853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:49823 (local address 127.16.62.126:40383)
0926 23:16:23.947018 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:23.947022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:23.948179 (+ 1157us) server_negotiation.cc:407] Connection header received
0926 23:16:23.949936 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:23.949940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:23.950006 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:23.950091 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:23.951758 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.952628 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.954702 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.955122 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.955219 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:23.958537 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:23.958542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:23.958545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:23.958575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:23.960946 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:23.961592 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:23.961597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:23.961599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:23.961735 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:23.962132 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:23.962135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:23.962137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:23.962410 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:23.962539 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:23.963121 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 23:16:23.963293 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:16:23.964504 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:23.947368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:40383 (local address 127.16.62.67:49823)
0926 23:16:23.948020 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:23.948055 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:23.949688 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:23.950294 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:23.950301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:23.950816 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:23.951595 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:23.951617 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.952774 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.952779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:23.954564 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:23.954582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.955299 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.955304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:23.955479 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:23.956052 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:23.956073 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:23.958363 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:23.961057 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:23.961063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:23.961074 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:23.961465 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:23.961849 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:23.961851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:23.961854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:23.962037 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:23.962513 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:23.962518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:23.962809 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 23:16:23.963014 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 23:16:23.963372 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":248,"threads_started":1}
I20260926 23:16:23.965770 17665 heartbeater.cc:347] Connected to a master server at 127.16.62.126:40383
I20260926 23:16:23.966025 17665 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:23.966665 17665 heartbeater.cc:511] Master 127.16.62.126:40383 requested a full tablet report, sending...
I20260926 23:16:23.967996 17209 ts_manager.cc:194] Registered new tserver with Master: 03b5dd11f6b74dd1bbfbb0a924e2d2b9 (127.16.62.67:43587)
I20260926 23:16:23.968834 17209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:49823
I20260926 23:16:23.978801 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:16:23.997576 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:23.985486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.16.62.126:40383)
0926 23:16:23.985686 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:23.985691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:23.986245 (+ 554us) server_negotiation.cc:407] Connection header received
0926 23:16:23.986378 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:23.986381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:23.986441 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:23.986518 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:23.987856 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.988632 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.989883 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:23.990246 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:23.990334 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:23.992504 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:23.992509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:23.992512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:23.992540 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:23.995027 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:23.995746 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:23.995750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:23.995752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:23.995883 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:23.996285 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:23.996288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:23.996290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:23.996627 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:23.996800 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:23.997170 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 23:16:23.997331 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 23:16:24.005271 17209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:16:24.012650 17209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:24.032506 17398 heartbeater.cc:503] Master 127.16.62.126:40383 was elected leader, sending a full tablet report...
I20260926 23:16:24.049117 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.025915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:43587 (local address 127.0.0.1:39838)
0926 23:16:24.037277 (+ 11362us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:24.037323 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:24.037569 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:24.039931 (+ 2362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:24.039943 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:24.039988 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:24.040609 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.040629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.043093 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.043102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.045735 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.045748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.047970 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.047977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.048273 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.048338 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 23:16:24.048455 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 23:16:24.048650 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11119,"thread_start_us":17393,"threads_started":1}
I20260926 23:16:24.049188 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.026153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.16.62.67:43587)
0926 23:16:24.039040 (+ 12887us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.039053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.039106 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:16:24.039238 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.039251 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.039597 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.040884 (+ 1287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:24.041202 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.042893 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.045907 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.047405 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.047545 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.047830 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.048669 (+ 839us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.048845 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12664,"thread_start_us":12467,"threads_started":1}
I20260926 23:16:24.049836 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.026162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:39089 (local address 127.0.0.1:53130)
0926 23:16:24.041510 (+ 15348us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:24.041530 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:24.041747 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:24.042678 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:24.042683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:24.042708 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:24.043231 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.043253 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.045507 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.045512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.046905 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.046916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.048987 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.048993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.049169 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.049203 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 23:16:24.049309 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 23:16:24.049443 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15043,"mutex_wait_us":6,"thread_start_us":10695,"threads_started":1}
I20260926 23:16:24.050071 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.026280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.16.62.66:39089)
0926 23:16:24.038199 (+ 11919us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.038213 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.041637 (+ 3424us) server_negotiation.cc:407] Connection header received
0926 23:16:24.041971 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.041981 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.042203 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.042381 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:24.043502 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.045366 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.047052 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.048552 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.048675 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.048848 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.049515 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.049701 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11698,"thread_start_us":11938,"threads_started":1}
I20260926 23:16:24.051627 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.037785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:41971 (local address 127.0.0.1:44202)
0926 23:16:24.043714 (+ 5929us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:24.043733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:24.043858 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:24.044833 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:24.044855 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:24.044881 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:24.045313 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.045329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.047201 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.047206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.049291 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.049304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.050927 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.050933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.051085 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.051123 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:16:24.051201 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 23:16:24.051322 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5767,"thread_start_us":3538,"threads_started":1}
I20260926 23:16:24.051736 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.038107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.16.62.65:41971)
0926 23:16:24.042306 (+ 4199us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.042312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.043993 (+ 1681us) server_negotiation.cc:407] Connection header received
0926 23:16:24.044113 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.044121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.044364 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.044566 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:24.045563 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.047060 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.049447 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.050737 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.050887 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.051090 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.051251 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.051384 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4065,"thread_start_us":3892,"threads_started":1}
I20260926 23:16:24.053632 17600 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4b255fa3584c43b0177555a5ebfcfb (DEFAULT_TABLE table=test-table [id=bb86844abb844559b09e5e697afc4e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:24.054685 17333 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4b255fa3584c43b0177555a5ebfcfb (DEFAULT_TABLE table=test-table [id=bb86844abb844559b09e5e697afc4e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:24.055225 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4b255fa3584c43b0177555a5ebfcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:24.055950 17333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4b255fa3584c43b0177555a5ebfcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:24.056277 17467 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4b255fa3584c43b0177555a5ebfcfb (DEFAULT_TABLE table=test-table [id=bb86844abb844559b09e5e697afc4e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:24.057624 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4b255fa3584c43b0177555a5ebfcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:24.067374 17683 tablet_bootstrap.cc:492] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85: Bootstrap starting.
I20260926 23:16:24.070946 17683 tablet_bootstrap.cc:654] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:24.071957 17683 log.cc:824] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:24.072566 17685 tablet_bootstrap.cc:492] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b: Bootstrap starting.
I20260926 23:16:24.074577 17684 tablet_bootstrap.cc:492] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9: Bootstrap starting.
I20260926 23:16:24.076491 17683 tablet_bootstrap.cc:492] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85: No bootstrap required, opened a new log
I20260926 23:16:24.076649 17685 tablet_bootstrap.cc:654] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:24.076830 17683 ts_tablet_manager.cc:1397] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 23:16:24.077833 17685 log.cc:824] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:24.078377 17684 tablet_bootstrap.cc:654] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:24.079844 17684 log.cc:824] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:24.083907 17684 tablet_bootstrap.cc:492] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9: No bootstrap required, opened a new log
I20260926 23:16:24.084285 17684 ts_tablet_manager.cc:1397] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260926 23:16:24.089408 17685 tablet_bootstrap.cc:492] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b: No bootstrap required, opened a new log
I20260926 23:16:24.089187 17683 raft_consensus.cc:348] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } }
I20260926 23:16:24.089831 17683 raft_consensus.cc:374] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:24.089892 17685 ts_tablet_manager.cc:1397] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20260926 23:16:24.090050 17683 raft_consensus.cc:729] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f0f8c211064236ae97e775939dbc85, State: Initialized, Role: FOLLOWER
I20260926 23:16:24.090865 17683 consensus_queue.cc:260] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } }
I20260926 23:16:24.098524 17683 ts_tablet_manager.cc:1428] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85: Time spent starting tablet: real 0.021s user 0.007s sys 0.013s
I20260926 23:16:24.098764 17684 raft_consensus.cc:348] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } }
I20260926 23:16:24.099174 17684 raft_consensus.cc:374] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:24.099268 17684 raft_consensus.cc:729] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b5dd11f6b74dd1bbfbb0a924e2d2b9, State: Initialized, Role: FOLLOWER
I20260926 23:16:24.099845 17684 consensus_queue.cc:260] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } }
I20260926 23:16:24.101110 17665 heartbeater.cc:503] Master 127.16.62.126:40383 was elected leader, sending a full tablet report...
I20260926 23:16:24.101966 17685 raft_consensus.cc:348] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } }
I20260926 23:16:24.102449 17685 raft_consensus.cc:374] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:24.102591 17685 raft_consensus.cc:729] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371567e73d114132a4932c9ac1e7521b, State: Initialized, Role: FOLLOWER
I20260926 23:16:24.102893 17684 ts_tablet_manager.cc:1428] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9: Time spent starting tablet: real 0.019s user 0.008s sys 0.008s
I20260926 23:16:24.103184 17685 consensus_queue.cc:260] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } }
I20260926 23:16:24.104267 17532 heartbeater.cc:503] Master 127.16.62.126:40383 was elected leader, sending a full tablet report...
I20260926 23:16:24.106159 17685 ts_tablet_manager.cc:1428] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260926 23:16:24.143272 17690 raft_consensus.cc:482] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:24.143584 17690 raft_consensus.cc:504] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } }
I20260926 23:16:24.145563 17690 leader_election.cc:288] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371567e73d114132a4932c9ac1e7521b (127.16.62.66:39089), e6f0f8c211064236ae97e775939dbc85 (127.16.62.65:41971)
I20260926 23:16:24.152122 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.145898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:39089 (local address 127.16.62.67:45109)
0926 23:16:24.146117 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:24.146138 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:24.146267 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:24.146755 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:24.146762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:24.146793 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:24.147007 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.147019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.148398 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.148401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.150302 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.150315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.151601 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.151605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.151739 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.151763 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 23:16:24.151832 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 23:16:24.151927 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":23}
I20260926 23:16:24.152290 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.146109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:45109 (local address 127.16.62.66:39089)
0926 23:16:24.146335 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.146341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.146364 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:16:24.146440 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.146447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.146533 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.146648 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:24.147194 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.148283 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.150458 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.151464 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.151629 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.151706 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.151880 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.152006 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:16:24.152979 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.146426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:33057 (local address 127.16.62.65:41971)
0926 23:16:24.146642 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.146650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.147160 (+ 510us) server_negotiation.cc:407] Connection header received
0926 23:16:24.147247 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.147252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.147333 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.147433 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:24.147881 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.148772 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.150916 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.152106 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.152277 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.152365 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.152624 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.152772 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 23:16:24.153082 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.146294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:41971 (local address 127.16.62.67:33057)
0926 23:16:24.146989 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:24.147005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:24.147123 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:24.147446 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:24.147449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:24.147466 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:24.147731 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.147738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.148910 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.148915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.150767 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:24.150777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.152321 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.152326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:24.152470 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.152500 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:16:24.152593 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 23:16:24.152722 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":4,"thread_start_us":401,"threads_started":1}
I20260926 23:16:24.153657 17487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4b255fa3584c43b0177555a5ebfcfb" candidate_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371567e73d114132a4932c9ac1e7521b" is_pre_election: true
I20260926 23:16:24.154140 17353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4b255fa3584c43b0177555a5ebfcfb" candidate_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f0f8c211064236ae97e775939dbc85" is_pre_election: true
I20260926 23:16:24.154650 17487 raft_consensus.cc:2454] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b5dd11f6b74dd1bbfbb0a924e2d2b9 in term 0.
I20260926 23:16:24.154739 17353 raft_consensus.cc:2454] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b5dd11f6b74dd1bbfbb0a924e2d2b9 in term 0.
I20260926 23:16:24.155400 17555 leader_election.cc:302] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b5dd11f6b74dd1bbfbb0a924e2d2b9, 371567e73d114132a4932c9ac1e7521b; no voters:
I20260926 23:16:24.155788 17690 raft_consensus.cc:2781] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:24.155947 17690 raft_consensus.cc:482] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:24.156069 17690 raft_consensus.cc:3037] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:24.158347 17690 raft_consensus.cc:504] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } }
I20260926 23:16:24.158875 17690 leader_election.cc:288] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [CANDIDATE]: Term 1 election: Requested vote from peers 371567e73d114132a4932c9ac1e7521b (127.16.62.66:39089), e6f0f8c211064236ae97e775939dbc85 (127.16.62.65:41971)
I20260926 23:16:24.159358 17487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4b255fa3584c43b0177555a5ebfcfb" candidate_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371567e73d114132a4932c9ac1e7521b"
I20260926 23:16:24.159447 17353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4b255fa3584c43b0177555a5ebfcfb" candidate_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f0f8c211064236ae97e775939dbc85"
I20260926 23:16:24.159560 17487 raft_consensus.cc:3037] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:24.159667 17353 raft_consensus.cc:3037] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:24.161829 17353 raft_consensus.cc:2454] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b5dd11f6b74dd1bbfbb0a924e2d2b9 in term 1.
I20260926 23:16:24.161868 17487 raft_consensus.cc:2454] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b5dd11f6b74dd1bbfbb0a924e2d2b9 in term 1.
I20260926 23:16:24.162209 17555 leader_election.cc:302] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b5dd11f6b74dd1bbfbb0a924e2d2b9, e6f0f8c211064236ae97e775939dbc85; no voters:
I20260926 23:16:24.162510 17690 raft_consensus.cc:2781] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:24.162942 17690 raft_consensus.cc:686] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [term 1 LEADER]: Becoming Leader. State: Replica: 03b5dd11f6b74dd1bbfbb0a924e2d2b9, State: Running, Role: LEADER
I20260926 23:16:24.163394 17690 consensus_queue.cc:237] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } }
I20260926 23:16:24.168646 17209 catalog_manager.cc:5495] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03b5dd11f6b74dd1bbfbb0a924e2d2b9 (127.16.62.67). New cstate: current_term: 1 leader_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b5dd11f6b74dd1bbfbb0a924e2d2b9" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371567e73d114132a4932c9ac1e7521b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 39089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f0f8c211064236ae97e775939dbc85" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41971 } health_report { overall_health: UNKNOWN } } }
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:24.193858 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.179711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.16.62.65:41971)
0926 23:16:24.179871 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.179875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.180498 (+ 623us) server_negotiation.cc:407] Connection header received
0926 23:16:24.180713 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.180720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.180833 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.180941 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:24.182541 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.183252 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.184558 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.184969 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.185109 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.188951 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:24.188964 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:24.188978 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:24.189008 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:24.191559 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.192130 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.192179 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.192186 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.192321 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.192690 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.192693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.192695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.193020 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:24.193225 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.193450 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.193589 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 23:16:24.196014 17666 tablet.cc:2369] T 3d4b255fa3584c43b0177555a5ebfcfb P 03b5dd11f6b74dd1bbfbb0a924e2d2b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:16:24.217330 17533 tablet.cc:2369] T 3d4b255fa3584c43b0177555a5ebfcfb P 371567e73d114132a4932c9ac1e7521b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:24.237659 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.224914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.16.62.65:41971)
0926 23:16:24.225146 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.225152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.225355 (+ 203us) server_negotiation.cc:407] Connection header received
0926 23:16:24.225498 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.225502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.225564 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.225645 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:24.226835 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.227542 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.228805 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.229196 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.229290 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.232811 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:24.232815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:24.232818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:24.232847 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:24.235114 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.235678 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.235682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.235684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.235817 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.236292 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.236298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.236304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.236558 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:24.236877 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.237125 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.237335 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 23:16:24.249715 17399 tablet.cc:2369] T 3d4b255fa3584c43b0177555a5ebfcfb P e6f0f8c211064236ae97e775939dbc85: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:24.284803 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.267268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.16.62.65:41971)
0926 23:16:24.267478 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.267484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.267638 (+ 154us) server_negotiation.cc:407] Connection header received
0926 23:16:24.267780 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.267784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.267843 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.267922 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:24.269173 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.269829 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.271352 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.271996 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.272089 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.278047 (+ 5958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:24.278052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:24.278055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:24.278087 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:24.281787 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.282557 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.282561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.282563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.282741 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.283087 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.283090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.283091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.283463 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:24.283723 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.283902 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.284195 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 23:16:24.287237 17313 tablet_service.cc:2908] Rejecting scan request for tablet 3d4b255fa3584c43b0177555a5ebfcfb: Uninitialized: clean time has not yet been initialized
I20260926 23:16:24.291306 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17270
I20260926 23:16:24.305400 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.291990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.16.62.65:41971)
0926 23:16:24.292202 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.292211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.292504 (+ 293us) server_negotiation.cc:407] Connection header received
0926 23:16:24.292623 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.292627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.292691 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.292767 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:24.294290 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.294934 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.296503 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.297130 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.297272 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.299850 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:24.299855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:24.299858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:24.299885 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:24.303248 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.303771 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.303775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.303777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.303881 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.304208 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.304216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.304219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.304477 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:24.304676 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.304882 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.305030 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:16:24.306077 17393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:24.396015 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 17270
I20260926 23:16:24.421337 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17404
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:16:24.436626 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.422311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.16.62.66:39089)
0926 23:16:24.422515 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.422522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.422542 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:16:24.422608 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.422613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.422701 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.422809 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:24.424017 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.425179 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.426432 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.426994 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.427159 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.430970 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:24.430983 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:24.430998 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:24.431031 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:24.433774 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.434409 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.434417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.434419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.434579 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.434998 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.435005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.435008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.435353 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:24.435672 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.435947 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.436345 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:16:24.437367 17527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:24.524621 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 17404
I20260926 23:16:24.540911 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17537
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:16:24.555234 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:24.541778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.16.62.67:43587)
0926 23:16:24.541945 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:24.541949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:24.541966 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:16:24.542020 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:24.542024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:24.542093 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:24.542178 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:24.543393 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.544110 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.545401 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:24.545815 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:24.545916 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:24.549908 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:24.549920 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:24.549935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:24.549964 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:24.552701 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.553333 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.553341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.553343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.553466 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:24.553821 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:24.553827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:24.553830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:24.554156 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:24.554435 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:24.554705 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 23:16:24.554936 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:16:24.555881 17660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:24.645968 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 17537
I20260926 23:16:24.664700 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17177
I20260926 23:16:24.665917 17239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:24.770928 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 17177
2026-09-26T23:16:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4090 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17712](info): set up 2 sockets
Sep 26 23:16:24 dist-test-slave-vcv1 krb5kdc[17712](info): commencing operation
krb5kdc: starting...
W20260926 23:16:25.844937 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:16:25 dist-test-slave-vcv1 krb5kdc[17712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:16:26.020845 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44059
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:44059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:26.329720 17725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:26.330063 17725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:26.330236 17725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:26.337435 17725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:26.337577 17725 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 23:16:26.337682 17725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:26.337801 17725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:26.337885 17725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:26.337973 17725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:26.347625 17725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:44059
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:26.348906 17725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:26.350260 17725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:26.361522 17730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:26.361557 17733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:26.361589 17731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:26.362509 17725 server_base.cc:1034] running on GCE node
I20260926 23:16:26.363230 17725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:26.364607 17725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:26.365854 17725 hybrid_clock.cc:648] HybridClock initialized: now 1790464586365811 us; error 46 us; skew 500 ppm
I20260926 23:16:26.366176 17725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:26 dist-test-slave-vcv1 krb5kdc[17712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464586, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:26.369211 17725 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:16:26.370602 17725 webserver.cc:460] Webserver started at http://127.16.62.126:44633/ using document root <none> and password file <none>
I20260926 23:16:26.371256 17725 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:26.371377 17725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:26.371694 17725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:26.374543 17725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "9e9829ad0470494facef0e31cc7abb14"
format_stamp: "Formatted at 2026-09-26 23:16:26 on dist-test-slave-vcv1"
server_key: "77e5f8fa99ccae390dfcdd950d6a2e27"
server_key_iv: "fe4b701e409e86e265a00d897becf8de"
server_key_version: "encryptionkey@0"
I20260926 23:16:26.375290 17725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "9e9829ad0470494facef0e31cc7abb14"
format_stamp: "Formatted at 2026-09-26 23:16:26 on dist-test-slave-vcv1"
server_key: "77e5f8fa99ccae390dfcdd950d6a2e27"
server_key_iv: "fe4b701e409e86e265a00d897becf8de"
server_key_version: "encryptionkey@0"
I20260926 23:16:26.380100 17725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 23:16:26.383371 17740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:26.384586 17725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:26.384775 17725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "9e9829ad0470494facef0e31cc7abb14"
format_stamp: "Formatted at 2026-09-26 23:16:26 on dist-test-slave-vcv1"
server_key: "77e5f8fa99ccae390dfcdd950d6a2e27"
server_key_iv: "fe4b701e409e86e265a00d897becf8de"
server_key_version: "encryptionkey@0"
I20260926 23:16:26.384984 17725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:26.403041 17725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:26.406662 17725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:26.406920 17725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:26.427567 17725 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:44059
I20260926 23:16:26.427585 17801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:44059 every 8 connection(s)
I20260926 23:16:26.428913 17725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:26.432374 17802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:26.437099 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 17725
I20260926 23:16:26.437315 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:26.438107 16633 external_mini_cluster.cc:1428] Setting key 5dcfd2d0b3e6841327d6f7bf2740040d
I20260926 23:16:26.443161 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14: Bootstrap starting.
I20260926 23:16:26.446650 17802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:26.447842 17802 log.cc:824] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:26.450987 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14: No bootstrap required, opened a new log
Sep 26 23:16:26 dist-test-slave-vcv1 krb5kdc[17712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:26.458524 17802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9829ad0470494facef0e31cc7abb14" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44059 } }
I20260926 23:16:26.458861 17802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:26.458979 17802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9829ad0470494facef0e31cc7abb14, State: Initialized, Role: FOLLOWER
I20260926 23:16:26.459591 17802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9829ad0470494facef0e31cc7abb14" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44059 } }
I20260926 23:16:26.459982 17802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:26.460121 17802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:26.460327 17802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:26.461390 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:26.440281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.16.62.126:44059)
0926 23:16:26.441089 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:26.441100 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:26.441141 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:16:26.442766 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:26.442786 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:26.443136 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:26.443485 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:26.445218 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:26.446504 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:26.448384 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:26.449410 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:26.449540 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:26.454173 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:26.454181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:26.454193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:26.454229 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:26.457318 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:26.457987 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:26.457993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:26.457998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:26.458158 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:26.458536 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:26.458540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:26.458542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:26.459186 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:26.459480 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:26.459918 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 23:16:26.460383 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":349,"threads_started":1}
I20260926 23:16:26.462611 17802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9829ad0470494facef0e31cc7abb14" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44059 } }
I20260926 23:16:26.463092 17802 leader_election.cc:302] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e9829ad0470494facef0e31cc7abb14; no voters:
I20260926 23:16:26.463706 17802 leader_election.cc:288] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:26.463826 17807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:26.466120 17802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:26.466630 17807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [term 1 LEADER]: Becoming Leader. State: Replica: 9e9829ad0470494facef0e31cc7abb14, State: Running, Role: LEADER
I20260926 23:16:26.467218 17807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9829ad0470494facef0e31cc7abb14" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44059 } }
I20260926 23:16:26.470232 17809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e9829ad0470494facef0e31cc7abb14. Latest consensus state: current_term: 1 leader_uuid: "9e9829ad0470494facef0e31cc7abb14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9829ad0470494facef0e31cc7abb14" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44059 } } }
I20260926 23:16:26.470278 17808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e9829ad0470494facef0e31cc7abb14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9829ad0470494facef0e31cc7abb14" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44059 } } }
I20260926 23:16:26.470623 17808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:26.470623 17809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:26.471752 17817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:26.476076 17817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:26.483673 17817 catalog_manager.cc:1347] Generated new cluster ID: 04358631ea814e03952f012620acb63c
I20260926 23:16:26.483778 17817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:26.495040 17817 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:26.496919 17817 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:26.508630 17817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14: Generated new TSK 0
I20260926 23:16:26.509647 17817 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:16:26.555879 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44059
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:26.875496 17828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:26.875816 17828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:26.875988 17828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:26.883684 17828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:26.883785 17828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:26.883958 17828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:16:26.884080 17828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:16:26.894178 17828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.62.126:44059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:26.895469 17828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:26.896868 17828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:26.909618 17833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:26.909574 17836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:26.909644 17834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:26.910624 17828 server_base.cc:1034] running on GCE node
I20260926 23:16:26.911304 17828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:26.912304 17828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:26.913575 17828 hybrid_clock.cc:648] HybridClock initialized: now 1790464586913489 us; error 54 us; skew 500 ppm
I20260926 23:16:26.913904 17828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:26 dist-test-slave-vcv1 krb5kdc[17712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464586, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:26.917001 17828 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:16:26.918357 17828 webserver.cc:460] Webserver started at http://127.16.62.65:39717/ using document root <none> and password file <none>
I20260926 23:16:26.919049 17828 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:26.919175 17828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:26.919486 17828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:26.922416 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7"
format_stamp: "Formatted at 2026-09-26 23:16:26 on dist-test-slave-vcv1"
server_key: "3eea8965707d5089ade4f923ac7e1e4e"
server_key_iv: "745a170440857190c2ece2a10514386d"
server_key_version: "encryptionkey@0"
I20260926 23:16:26.923210 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7"
format_stamp: "Formatted at 2026-09-26 23:16:26 on dist-test-slave-vcv1"
server_key: "3eea8965707d5089ade4f923ac7e1e4e"
server_key_iv: "745a170440857190c2ece2a10514386d"
server_key_version: "encryptionkey@0"
I20260926 23:16:26.928174 17828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:16:26.931452 17843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:26.932799 17828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:26.932983 17828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7"
format_stamp: "Formatted at 2026-09-26 23:16:26 on dist-test-slave-vcv1"
server_key: "3eea8965707d5089ade4f923ac7e1e4e"
server_key_iv: "745a170440857190c2ece2a10514386d"
server_key_version: "encryptionkey@0"
I20260926 23:16:26.933183 17828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:26.968541 17828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:26.972955 17828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:26.973227 17828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:26.975716 17828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:26.975880 17828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:26.976037 17828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:26.976112 17828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:16:26 dist-test-slave-vcv1 krb5kdc[17712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464586, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:27.000720 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:26.976471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:56877 (local address 127.16.62.126:44059)
0926 23:16:26.977136 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:26.977144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:26.978154 (+ 1010us) server_negotiation.cc:407] Connection header received
0926 23:16:26.980592 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:26.980597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:26.980680 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:26.980804 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:26.983446 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:26.984413 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:26.987578 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:26.988216 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:26.988336 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:26.993217 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:26.993222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:26.993227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:26.993297 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:26.996619 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:26.997817 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:26.997826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:26.997830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:26.997977 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:26.998567 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:26.998575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:26.998578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:26.998864 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:26.998971 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:27.000030 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0926 23:16:27.000274 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":432,"threads_started":1}
I20260926 23:16:27.002564 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:26.977170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44059 (local address 127.16.62.65:56877)
0926 23:16:26.977973 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:26.978030 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:26.980306 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:26.981184 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:26.981199 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:26.982184 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:26.983216 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:26.983246 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:26.984598 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:26.984606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:26.987408 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:26.987428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:26.988490 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:26.988500 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:26.988854 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:26.989682 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:26.989720 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:26.992976 (+ 3256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:26.996796 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:26.996807 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:26.996827 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:26.997689 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:26.998119 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:26.998127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:26.998133 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:26.998440 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:26.999018 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:26.999030 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:26.999584 (+ 554us) client_negotiation.cc:769] Sending connection context
0926 23:16:26.999949 (+ 365us) client_negotiation.cc:240] Negotiation successful
0926 23:16:27.000689 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":364,"threads_started":1}
I20260926 23:16:27.029408 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:27.015349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:56741 (local address 127.16.62.126:44059)
0926 23:16:27.015530 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:27.015539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:27.016337 (+ 798us) server_negotiation.cc:407] Connection header received
0926 23:16:27.016581 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:27.016589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:27.016690 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:27.016820 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:27.017987 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.018868 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.021060 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.021458 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.021628 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:27.023497 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:27.023503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:27.023508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:27.023553 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:27.026657 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:27.027435 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:27.027441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:27.027445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:27.027578 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:27.027930 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:27.027936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:27.027939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:27.028280 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:27.028417 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:27.028842 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 23:16:27.029056 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:16:27.029635 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:27.015417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44059 (local address 127.16.62.65:56741)
0926 23:16:27.016206 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:27.016232 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:27.016411 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:27.016846 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:27.016850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:27.017527 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:27.017836 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:27.017848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.019000 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.019009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:27.020918 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:27.020933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.021673 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.021679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:27.021842 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:27.022547 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:27.022573 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:27.023325 (+ 752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:27.026777 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:27.026782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:27.026786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:27.027266 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:27.027685 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:27.027690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:27.027697 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:27.027819 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:27.028382 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:27.028387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:27.028594 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 23:16:27.028742 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 23:16:27.029056 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":361,"threads_started":1}
I20260926 23:16:27.030639 17828 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:36453
I20260926 23:16:27.030660 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:36453 every 8 connection(s)
I20260926 23:16:27.031932 17828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:16:27.034236 17964 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44059
I20260926 23:16:27.034349 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 17828
I20260926 23:16:27.034425 17964 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:27.034544 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:16:27.034961 17964 heartbeater.cc:511] Master 127.16.62.126:44059 requested a full tablet report, sending...
I20260926 23:16:27.035244 16633 external_mini_cluster.cc:1428] Setting key 14c0a34f5a577aa387ced30986543464
I20260926 23:16:27.037074 17757 ts_manager.cc:194] Registered new tserver with Master: 3f04a4f42b1b4f0dad87229b22a8a3b7 (127.16.62.65:36453)
I20260926 23:16:27.039168 17757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:56877
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:16:27.075518 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44059
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:27.383716 17968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:27.383980 17968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:27.384074 17968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:27.391175 17968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:27.391275 17968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:27.391355 17968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:16:27.391435 17968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:16:27.400808 17968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.62.126:44059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:27.401953 17968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:27.403203 17968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:27.416332 17973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:27.416378 17976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:27.416430 17974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:27.417447 17968 server_base.cc:1034] running on GCE node
I20260926 23:16:27.418031 17968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:27.418884 17968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:27.420120 17968 hybrid_clock.cc:648] HybridClock initialized: now 1790464587420065 us; error 33 us; skew 500 ppm
I20260926 23:16:27.420472 17968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:27 dist-test-slave-vcv1 krb5kdc[17712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:27.430634 17968 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:16:27.432412 17968 webserver.cc:460] Webserver started at http://127.16.62.66:43121/ using document root <none> and password file <none>
I20260926 23:16:27.433091 17968 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:27.433200 17968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:27.433522 17968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:27.436363 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "45f26718e525426cbe27bf48f96a58cb"
format_stamp: "Formatted at 2026-09-26 23:16:27 on dist-test-slave-vcv1"
server_key: "79b870acf596e729e8988b56b00261b7"
server_key_iv: "0e4b50deb4e02deaa91d26e2763ed839"
server_key_version: "encryptionkey@0"
I20260926 23:16:27.437116 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "45f26718e525426cbe27bf48f96a58cb"
format_stamp: "Formatted at 2026-09-26 23:16:27 on dist-test-slave-vcv1"
server_key: "79b870acf596e729e8988b56b00261b7"
server_key_iv: "0e4b50deb4e02deaa91d26e2763ed839"
server_key_version: "encryptionkey@0"
I20260926 23:16:27.442090 17968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:16:27.445263 17983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:27.446518 17968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 23:16:27.446662 17968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "45f26718e525426cbe27bf48f96a58cb"
format_stamp: "Formatted at 2026-09-26 23:16:27 on dist-test-slave-vcv1"
server_key: "79b870acf596e729e8988b56b00261b7"
server_key_iv: "0e4b50deb4e02deaa91d26e2763ed839"
server_key_version: "encryptionkey@0"
I20260926 23:16:27.446925 17968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:27.476643 17968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:27.480434 17968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:27.480755 17968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:27.483186 17968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:27.483294 17968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:27.483438 17968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:27.483503 17968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:16:27 dist-test-slave-vcv1 krb5kdc[17712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:27.506436 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:27.483957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:49677 (local address 127.16.62.126:44059)
0926 23:16:27.484135 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:27.484141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:27.485661 (+ 1520us) server_negotiation.cc:407] Connection header received
0926 23:16:27.487887 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:27.487893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:27.487976 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:27.488084 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:27.490406 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.491841 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.494404 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.494896 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.495035 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:27.499191 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:27.499196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:27.499200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:27.499229 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:27.502277 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:27.503261 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:27.503269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:27.503274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:27.503417 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:27.504023 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:27.504030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:27.504035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:27.504520 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:27.504668 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:27.505615 (+ 947us) server_negotiation.cc:299] Negotiation successful
0926 23:16:27.505927 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:16:27.507222 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:27.484684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44059 (local address 127.16.62.66:49677)
0926 23:16:27.485505 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:27.485550 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:27.487576 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:27.488372 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:27.488382 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:27.489208 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:27.490190 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:27.490209 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.491999 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.492003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:27.494180 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:27.494194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.495145 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.495153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:27.495376 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:27.496059 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:27.496094 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:27.498969 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:27.502410 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:27.502417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:27.502436 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:27.503101 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:27.503536 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:27.503544 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:27.503550 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:27.503854 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:27.504677 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:27.504684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:27.505111 (+ 427us) client_negotiation.cc:769] Sending connection context
0926 23:16:27.505461 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 23:16:27.505881 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":386,"threads_started":1}
I20260926 23:16:27.530913 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:27.514424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:53117 (local address 127.16.62.126:44059)
0926 23:16:27.514652 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:27.514659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:27.515208 (+ 549us) server_negotiation.cc:407] Connection header received
0926 23:16:27.515400 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:27.515405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:27.515478 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:27.515587 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:27.517267 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.518510 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.521242 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.521699 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.521799 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:27.524386 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:27.524393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:27.524401 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:27.524443 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:27.527991 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:27.529000 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:27.529006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:27.529009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:27.529139 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:27.529526 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:27.529531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:27.529534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:27.529839 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:27.529991 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:27.530362 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 23:16:27.530570 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:16:27.531032 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:27.513973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44059 (local address 127.16.62.66:53117)
0926 23:16:27.515122 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:27.515144 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:27.515268 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:27.515786 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:27.515795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:27.516549 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:27.517051 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:27.517065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.518721 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.518726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:27.521078 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:27.521095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.521901 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.521910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:27.522145 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:27.522807 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:27.522837 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:27.524116 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:27.528227 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:27.528233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:27.528237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:27.528821 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:27.529252 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:27.529256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:27.529260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:27.529384 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:27.529957 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:27.529962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:27.530130 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 23:16:27.530264 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 23:16:27.530579 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":792,"threads_started":1}
I20260926 23:16:27.539132 17968 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:46235
I20260926 23:16:27.539152 18102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:46235 every 8 connection(s)
I20260926 23:16:27.540366 17968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:16:27.541896 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 17968
I20260926 23:16:27.542124 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:16:27.542239 18103 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44059
I20260926 23:16:27.542455 18103 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:27.542719 16633 external_mini_cluster.cc:1428] Setting key 53925a86dfbccd03c2b2a17c9a284b9d
I20260926 23:16:27.543182 18103 heartbeater.cc:511] Master 127.16.62.126:44059 requested a full tablet report, sending...
I20260926 23:16:27.544598 17757 ts_manager.cc:194] Registered new tserver with Master: 45f26718e525426cbe27bf48f96a58cb (127.16.62.66:46235)
I20260926 23:16:27.545353 17757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:49677
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:16:27.581115 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44059
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:27.892350 18107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:27.892627 18107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:27.892796 18107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:27.899856 18107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:27.899953 18107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:27.900120 18107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:16:27.900271 18107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:16:27.909614 18107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.62.126:44059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:27.910832 18107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:27.912101 18107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:27.924546 18112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:27.924456 18113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:27.924510 18115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:27.925318 18107 server_base.cc:1034] running on GCE node
I20260926 23:16:27.925931 18107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:27.926769 18107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:27.927996 18107 hybrid_clock.cc:648] HybridClock initialized: now 1790464587927924 us; error 54 us; skew 500 ppm
I20260926 23:16:27.928323 18107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:27 dist-test-slave-vcv1 krb5kdc[17712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:27.931272 18107 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:16:27.932709 18107 webserver.cc:460] Webserver started at http://127.16.62.67:42107/ using document root <none> and password file <none>
I20260926 23:16:27.933377 18107 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:27.933508 18107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:27.933828 18107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:27.936769 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "223677f3c68e4a7bb8668f802a45d248"
format_stamp: "Formatted at 2026-09-26 23:16:27 on dist-test-slave-vcv1"
server_key: "be0bb8092bc6288701988edde0f08bdd"
server_key_iv: "071a834d8b6b5c483beb2d0e39c3c0b6"
server_key_version: "encryptionkey@0"
I20260926 23:16:27.937529 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "223677f3c68e4a7bb8668f802a45d248"
format_stamp: "Formatted at 2026-09-26 23:16:27 on dist-test-slave-vcv1"
server_key: "be0bb8092bc6288701988edde0f08bdd"
server_key_iv: "071a834d8b6b5c483beb2d0e39c3c0b6"
server_key_version: "encryptionkey@0"
I20260926 23:16:27.942366 18107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:16:27.945663 18122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:27.947080 18107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:16:27.947260 18107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "223677f3c68e4a7bb8668f802a45d248"
format_stamp: "Formatted at 2026-09-26 23:16:27 on dist-test-slave-vcv1"
server_key: "be0bb8092bc6288701988edde0f08bdd"
server_key_iv: "071a834d8b6b5c483beb2d0e39c3c0b6"
server_key_version: "encryptionkey@0"
I20260926 23:16:27.947472 18107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:27.969553 18107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:27.973241 18107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:27.973495 18107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:27.976028 18107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:27.976157 18107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:27.976310 18107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:27.976389 18107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:16:27 dist-test-slave-vcv1 krb5kdc[17712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:27.998323 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:27.977290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44059 (local address 127.16.62.67:44505)
0926 23:16:27.978131 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:27.978204 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:27.980914 (+ 2710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:27.981706 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:27.981714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:27.982474 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:27.983434 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:27.983466 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.984941 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.984947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:27.986175 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:27.986186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.986975 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.986982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:27.987177 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:27.987838 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:27.987878 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:27.990731 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:27.994596 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:27.994604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:27.994618 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:27.995012 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:27.995374 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:27.995380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:27.995383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:27.995558 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:27.996087 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:27.996095 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:27.996448 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 23:16:27.996809 (+ 361us) client_negotiation.cc:240] Negotiation successful
0926 23:16:27.997251 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":346,"threads_started":1}
I20260926 23:16:27.998854 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:27.976719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:44505 (local address 127.16.62.126:44059)
0926 23:16:27.976907 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:27.976912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:27.978336 (+ 1424us) server_negotiation.cc:407] Connection header received
0926 23:16:27.981254 (+ 2918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:27.981261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:27.981344 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:27.981475 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:27.983679 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.984799 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.986321 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:27.986757 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:27.986892 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:27.991006 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:27.991014 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:27.991023 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:27.991054 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:27.994469 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:27.995135 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:27.995141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:27.995145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:27.995271 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:27.995670 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:27.995674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:27.995678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:27.995963 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:27.996125 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:27.998052 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0926 23:16:27.998352 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:16:28.029448 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.010146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:60743 (local address 127.16.62.126:44059)
0926 23:16:28.010350 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.010359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.012784 (+ 2425us) server_negotiation.cc:407] Connection header received
0926 23:16:28.012955 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.012964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.013071 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.013203 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:28.014110 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.015611 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.018106 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.018878 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.019041 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.021650 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:28.021660 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:28.021668 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:28.021716 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:28.025670 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:28.026730 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:28.026738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:28.026744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:28.026921 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:28.027349 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:28.027357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:28.027363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:28.027788 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:28.027966 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.028599 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.028899 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:16:28.032789 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.010234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44059 (local address 127.16.62.67:60743)
0926 23:16:28.012655 (+ 2421us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:28.012673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:28.012825 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:28.013227 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:28.013230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:28.013697 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:28.013969 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.013977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.015870 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.015877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.017911 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.017930 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.019132 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.019142 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.019402 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.020305 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:28.020327 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:28.021436 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:28.025821 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:28.025833 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:28.025839 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:28.026566 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:28.027062 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:28.027069 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:28.027073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:28.027221 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:28.027935 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:28.027943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:28.028245 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 23:16:28.028445 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 23:16:28.032282 (+ 3837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2274,"thread_start_us":2112,"threads_started":1}
I20260926 23:16:28.040690 18107 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:33961
I20260926 23:16:28.040707 18241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:33961 every 8 connection(s)
I20260926 23:16:28.041818 17964 heartbeater.cc:503] Master 127.16.62.126:44059 was elected leader, sending a full tablet report...
I20260926 23:16:28.041985 18107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:16:28.043962 18242 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44059
I20260926 23:16:28.044183 18242 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:28.044760 18242 heartbeater.cc:511] Master 127.16.62.126:44059 requested a full tablet report, sending...
I20260926 23:16:28.045779 17757 ts_manager.cc:194] Registered new tserver with Master: 223677f3c68e4a7bb8668f802a45d248 (127.16.62.67:33961)
I20260926 23:16:28.046535 17757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:44505
I20260926 23:16:28.047535 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 18107
I20260926 23:16:28.047708 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:16:28.048111 16633 external_mini_cluster.cc:1428] Setting key 9421922301ec02ad2bb2a4f7cadaa1f7
I20260926 23:16:28.050839 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:16:28 dist-test-slave-vcv1 krb5kdc[17712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:16:28 dist-test-slave-vcv1 krb5kdc[17712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:28.093906 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.080981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.16.62.126:44059)
0926 23:16:28.081200 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.081208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.081379 (+ 171us) server_negotiation.cc:407] Connection header received
0926 23:16:28.081590 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.081600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.081671 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.081779 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:28.083125 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.083854 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.085131 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.085557 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.085693 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.089424 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:28.089429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:28.089432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:28.089462 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:28.091752 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:28.092349 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:28.092352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:28.092355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:28.092460 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:28.092769 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:28.092772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:28.092774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:28.093012 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:28.093186 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.093470 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.093640 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:16:28.097797 17757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:16:28.104789 17757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:28.137846 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.116322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.16.62.65:36453)
0926 23:16:28.126548 (+ 10226us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.126561 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.129041 (+ 2480us) server_negotiation.cc:407] Connection header received
0926 23:16:28.129160 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.129168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.129442 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.130505 (+ 1063us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:28.130724 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.132319 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.135209 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.136745 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.136890 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.137097 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.137381 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.137536 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9994,"thread_start_us":10876,"threads_started":1}
I20260926 23:16:28.137913 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.116128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:36453 (local address 127.0.0.1:40534)
0926 23:16:28.128850 (+ 12722us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:28.128868 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:28.128984 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:28.129700 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:28.129708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:28.129740 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:28.130277 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.130288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.132451 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.132459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.135080 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.135091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.136946 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.136952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.137144 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.137195 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 23:16:28.137341 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 23:16:28.137521 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12560,"mutex_wait_us":61,"thread_start_us":12420,"threads_started":1}
I20260926 23:16:28.139123 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.129171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.16.62.66:46235)
0926 23:16:28.129705 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.129714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.131442 (+ 1728us) server_negotiation.cc:407] Connection header received
0926 23:16:28.131621 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.131626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.131797 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.131974 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:28.132570 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.133583 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.136720 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.138073 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.138214 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.138413 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.138671 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.138841 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":288,"threads_started":1}
I20260926 23:16:28.139287 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.129751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:46235 (local address 127.0.0.1:43946)
0926 23:16:28.131324 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:28.131338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:28.131467 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:28.131961 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:28.131966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:28.131988 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:28.132328 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.132337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.133733 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.133739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.135800 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.135832 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.138294 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.138303 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.138476 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.138514 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:16:28.138608 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 23:16:28.138748 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1468,"mutex_wait_us":5,"thread_start_us":1354,"threads_started":1}
I20260926 23:16:28.140586 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.115958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:33961 (local address 127.0.0.1:33954)
0926 23:16:28.128109 (+ 12151us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:28.128205 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:28.128407 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:28.130283 (+ 1876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:28.130287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:28.130313 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:28.130747 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.130760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.132698 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.132715 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.136511 (+ 3796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.136526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.138767 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.138773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.138932 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.138964 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:16:28.139049 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 23:16:28.139177 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11803,"mutex_wait_us":1517,"thread_start_us":11626,"threads_started":1}
I20260926 23:16:28.141270 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.116199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.16.62.67:33961)
0926 23:16:28.129138 (+ 12939us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.129146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.129172 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:16:28.129245 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.129251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.129449 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.129942 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:28.130969 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.132570 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.136648 (+ 4078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.138289 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.138407 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.138633 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.140771 (+ 2138us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.140937 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12763,"thread_start_us":12579,"threads_started":1}
I20260926 23:16:28.142042 17894 tablet_service.cc:1467] Processing CreateTablet for tablet a2851c689a474b539f469b2c084c7505 (DEFAULT_TABLE table=test-table [id=da127ed1fac34eac9f4983815fafcf13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:28.142915 18033 tablet_service.cc:1467] Processing CreateTablet for tablet a2851c689a474b539f469b2c084c7505 (DEFAULT_TABLE table=test-table [id=da127ed1fac34eac9f4983815fafcf13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:28.143510 17894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2851c689a474b539f469b2c084c7505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:28.143982 18033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2851c689a474b539f469b2c084c7505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:28.144457 18172 tablet_service.cc:1467] Processing CreateTablet for tablet a2851c689a474b539f469b2c084c7505 (DEFAULT_TABLE table=test-table [id=da127ed1fac34eac9f4983815fafcf13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:28.145704 18172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2851c689a474b539f469b2c084c7505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:28.157747 18262 tablet_bootstrap.cc:492] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Bootstrap starting.
I20260926 23:16:28.160923 18261 tablet_bootstrap.cc:492] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: Bootstrap starting.
I20260926 23:16:28.162596 18262 tablet_bootstrap.cc:654] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:28.163746 18262 log.cc:824] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:28.165043 18261 tablet_bootstrap.cc:654] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:28.165031 18263 tablet_bootstrap.cc:492] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: Bootstrap starting.
I20260926 23:16:28.166205 18261 log.cc:824] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:28.166934 18262 tablet_bootstrap.cc:492] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: No bootstrap required, opened a new log
I20260926 23:16:28.167188 18262 ts_tablet_manager.cc:1397] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 23:16:28.168399 18263 tablet_bootstrap.cc:654] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:28.169291 18261 tablet_bootstrap.cc:492] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: No bootstrap required, opened a new log
I20260926 23:16:28.169552 18261 ts_tablet_manager.cc:1397] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 23:16:28.170181 18263 log.cc:824] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:28.175484 18263 tablet_bootstrap.cc:492] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: No bootstrap required, opened a new log
I20260926 23:16:28.175932 18263 ts_tablet_manager.cc:1397] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 23:16:28.180980 18261 raft_consensus.cc:348] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } }
I20260926 23:16:28.181533 18261 raft_consensus.cc:374] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:28.181718 18261 raft_consensus.cc:729] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f26718e525426cbe27bf48f96a58cb, State: Initialized, Role: FOLLOWER
I20260926 23:16:28.182447 18261 consensus_queue.cc:260] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } }
I20260926 23:16:28.183395 18262 raft_consensus.cc:348] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } }
I20260926 23:16:28.184046 18262 raft_consensus.cc:374] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:28.184103 18103 heartbeater.cc:503] Master 127.16.62.126:44059 was elected leader, sending a full tablet report...
I20260926 23:16:28.184237 18262 raft_consensus.cc:729] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f04a4f42b1b4f0dad87229b22a8a3b7, State: Initialized, Role: FOLLOWER
I20260926 23:16:28.184966 18261 ts_tablet_manager.cc:1428] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: Time spent starting tablet: real 0.015s user 0.008s sys 0.009s
I20260926 23:16:28.185292 18262 consensus_queue.cc:260] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } }
I20260926 23:16:28.189062 18262 ts_tablet_manager.cc:1428] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Time spent starting tablet: real 0.022s user 0.004s sys 0.019s
I20260926 23:16:28.193229 18263 raft_consensus.cc:348] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } }
I20260926 23:16:28.193637 18263 raft_consensus.cc:374] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:28.193770 18263 raft_consensus.cc:729] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223677f3c68e4a7bb8668f802a45d248, State: Initialized, Role: FOLLOWER
I20260926 23:16:28.194291 18263 consensus_queue.cc:260] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } }
I20260926 23:16:28.195730 18242 heartbeater.cc:503] Master 127.16.62.126:44059 was elected leader, sending a full tablet report...
I20260926 23:16:28.196080 18263 ts_tablet_manager.cc:1428] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: Time spent starting tablet: real 0.020s user 0.012s sys 0.009s
W20260926 23:16:28.286324 17965 tablet.cc:2369] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:16:28.292261 18104 tablet.cc:2369] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:16:28.293993 18243 tablet.cc:2369] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:16:28.342912 18268 raft_consensus.cc:482] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:28.343261 18268 raft_consensus.cc:504] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } }
I20260926 23:16:28.344972 18268 leader_election.cc:288] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 223677f3c68e4a7bb8668f802a45d248 (127.16.62.67:33961), 45f26718e525426cbe27bf48f96a58cb (127.16.62.66:46235)
I20260926 23:16:28.353350 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.345459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:36079 (local address 127.16.62.67:33961)
0926 23:16:28.345699 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.345706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.346621 (+ 915us) server_negotiation.cc:407] Connection header received
0926 23:16:28.346814 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.346819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.346898 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.346999 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:28.347813 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.348973 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.351423 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.352388 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.352579 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.352665 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.352958 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.353076 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:16:28.353365 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.345414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:60773 (local address 127.16.62.66:46235)
0926 23:16:28.345647 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.345654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.346622 (+ 968us) server_negotiation.cc:407] Connection header received
0926 23:16:28.346814 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.346819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.346898 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.346999 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:28.347813 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.348973 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.351499 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.352397 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.352579 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.352663 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.352988 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.353107 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:16:28.353547 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.345337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:33961 (local address 127.16.62.65:36079)
0926 23:16:28.346478 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:28.346502 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:28.346657 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:28.347028 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:28.347033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:28.347075 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:28.347632 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.347641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.349116 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.349120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.351270 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.351283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.352619 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.352626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.352788 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.352826 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:16:28.352933 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 23:16:28.353059 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":183,"spinlock_wait_cycles":2816,"thread_start_us":926,"threads_started":1}
I20260926 23:16:28.353935 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.345358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:46235 (local address 127.16.62.65:60773)
0926 23:16:28.346503 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:28.346518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:28.346666 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:28.347028 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:28.347033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:28.347066 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:28.347615 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.347632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.349115 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.349120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.351309 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.351358 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.352662 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.352668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.352841 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.352863 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:16:28.352965 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 23:16:28.353084 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":194,"spinlock_wait_cycles":42240,"thread_start_us":639,"threads_started":1}
I20260926 23:16:28.354514 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2851c689a474b539f469b2c084c7505" candidate_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223677f3c68e4a7bb8668f802a45d248" is_pre_election: true
I20260926 23:16:28.354938 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2851c689a474b539f469b2c084c7505" candidate_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f26718e525426cbe27bf48f96a58cb" is_pre_election: true
I20260926 23:16:28.355094 18196 raft_consensus.cc:2454] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f04a4f42b1b4f0dad87229b22a8a3b7 in term 0.
I20260926 23:16:28.355433 18058 raft_consensus.cc:2454] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f04a4f42b1b4f0dad87229b22a8a3b7 in term 0.
I20260926 23:16:28.355790 17846 leader_election.cc:302] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223677f3c68e4a7bb8668f802a45d248, 3f04a4f42b1b4f0dad87229b22a8a3b7; no voters:
I20260926 23:16:28.356244 18268 raft_consensus.cc:2781] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:28.356393 18268 raft_consensus.cc:482] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:28.356511 18268 raft_consensus.cc:3037] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:28.358888 18268 raft_consensus.cc:504] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } }
I20260926 23:16:28.359476 18268 leader_election.cc:288] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [CANDIDATE]: Term 1 election: Requested vote from peers 223677f3c68e4a7bb8668f802a45d248 (127.16.62.67:33961), 45f26718e525426cbe27bf48f96a58cb (127.16.62.66:46235)
I20260926 23:16:28.359877 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2851c689a474b539f469b2c084c7505" candidate_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f26718e525426cbe27bf48f96a58cb"
I20260926 23:16:28.359921 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2851c689a474b539f469b2c084c7505" candidate_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223677f3c68e4a7bb8668f802a45d248"
I20260926 23:16:28.360067 18058 raft_consensus.cc:3037] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:28.360109 18196 raft_consensus.cc:3037] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:28.362151 18058 raft_consensus.cc:2454] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f04a4f42b1b4f0dad87229b22a8a3b7 in term 1.
I20260926 23:16:28.362164 18196 raft_consensus.cc:2454] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f04a4f42b1b4f0dad87229b22a8a3b7 in term 1.
I20260926 23:16:28.362552 17844 leader_election.cc:302] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f04a4f42b1b4f0dad87229b22a8a3b7, 45f26718e525426cbe27bf48f96a58cb; no voters:
I20260926 23:16:28.362869 18268 raft_consensus.cc:2781] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:28.363262 18268 raft_consensus.cc:686] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 LEADER]: Becoming Leader. State: Replica: 3f04a4f42b1b4f0dad87229b22a8a3b7, State: Running, Role: LEADER
I20260926 23:16:28.363737 18268 consensus_queue.cc:237] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } }
I20260926 23:16:28.368028 17756 catalog_manager.cc:5495] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f04a4f42b1b4f0dad87229b22a8a3b7 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } health_report { overall_health: UNKNOWN } } }
I20260926 23:16:28.437495 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.431135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.16.62.65:36453)
0926 23:16:28.431367 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.431376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.431404 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:16:28.431795 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.431798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.431903 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.432015 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:28.432479 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.433165 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.435148 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.435525 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.435611 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.436208 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:28.436868 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:28.436967 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.437127 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.437199 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 23:16:28.446103 18196 raft_consensus.cc:1261] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [term 1 FOLLOWER]: Refusing update from remote peer 3f04a4f42b1b4f0dad87229b22a8a3b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:28.446102 18058 raft_consensus.cc:1261] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [term 1 FOLLOWER]: Refusing update from remote peer 3f04a4f42b1b4f0dad87229b22a8a3b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:28.449640 18272 consensus_queue.cc:1035] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260926 23:16:28.450114 18268 consensus_queue.cc:1035] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:28.482483 17756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:28.482806 17756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:28.486362 17756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14: Sending DeleteTablet for 3 replicas of tablet a2851c689a474b539f469b2c084c7505
I20260926 23:16:28.488205 18033 tablet_service.cc:1514] Processing DeleteTablet for tablet a2851c689a474b539f469b2c084c7505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:28 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:43946
I20260926 23:16:28.488960 17894 tablet_service.cc:1514] Processing DeleteTablet for tablet a2851c689a474b539f469b2c084c7505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:28 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:40534
I20260926 23:16:28.489800 18285 tablet_replica.cc:330] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: stopping tablet replica
I20260926 23:16:28.490003 18172 tablet_service.cc:1514] Processing DeleteTablet for tablet a2851c689a474b539f469b2c084c7505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:28 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:33954
I20260926 23:16:28.490384 17756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 23:16:28.490509 18285 raft_consensus.cc:2229] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 23:16:28.491264 17756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:28.491655 18285 raft_consensus.cc:2258] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:28.498370 18287 tablet_replica.cc:330] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: stopping tablet replica
I20260926 23:16:28.498876 18287 raft_consensus.cc:2229] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:28.499615 18287 raft_consensus.cc:2258] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:28.502156 18286 tablet_replica.cc:330] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: stopping tablet replica
I20260926 23:16:28.502590 18286 raft_consensus.cc:2229] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:16:28.502875 17894 tablet_service.cc:1467] Processing CreateTablet for tablet f0a9300ed78f4a0ab87ae13cf4b48b83 (DEFAULT_TABLE table=test-table [id=4231447c60834357aa8a3c379e45244d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:28.503319 18287 ts_tablet_manager.cc:1905] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:28.503342 18033 tablet_service.cc:1467] Processing CreateTablet for tablet f0a9300ed78f4a0ab87ae13cf4b48b83 (DEFAULT_TABLE table=test-table [id=4231447c60834357aa8a3c379e45244d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:28.503860 18033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a9300ed78f4a0ab87ae13cf4b48b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:28.504678 17894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a9300ed78f4a0ab87ae13cf4b48b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:28.505808 18286 raft_consensus.cc:2258] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:28.506883 18172 tablet_service.cc:1467] Processing CreateTablet for tablet f0a9300ed78f4a0ab87ae13cf4b48b83 (DEFAULT_TABLE table=test-table [id=4231447c60834357aa8a3c379e45244d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:28.507421 18172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a9300ed78f4a0ab87ae13cf4b48b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:28.511168 18285 ts_tablet_manager.cc:1905] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:28.514423 18287 ts_tablet_manager.cc:1918] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:16:28.514608 18287 log.cc:1196] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/a2851c689a474b539f469b2c084c7505
I20260926 23:16:28.515066 18287 ts_tablet_manager.cc:1939] T a2851c689a474b539f469b2c084c7505 P 223677f3c68e4a7bb8668f802a45d248: Deleting consensus metadata
I20260926 23:16:28.515887 18261 tablet_bootstrap.cc:492] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: Bootstrap starting.
I20260926 23:16:28.516247 18262 tablet_bootstrap.cc:492] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Bootstrap starting.
I20260926 23:16:28.516768 18286 ts_tablet_manager.cc:1905] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:28.516897 17743 catalog_manager.cc:4841] TS 223677f3c68e4a7bb8668f802a45d248 (127.16.62.67:33961): tablet a2851c689a474b539f469b2c084c7505 (table test-table [id=da127ed1fac34eac9f4983815fafcf13]) successfully deleted
I20260926 23:16:28.517616 18261 tablet_bootstrap.cc:654] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:28.517985 18262 tablet_bootstrap.cc:654] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:28.525907 18263 tablet_bootstrap.cc:492] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: Bootstrap starting.
I20260926 23:16:28.529649 18263 tablet_bootstrap.cc:654] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:28.534749 18285 ts_tablet_manager.cc:1918] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:16:28.535126 18285 log.cc:1196] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/a2851c689a474b539f469b2c084c7505
I20260926 23:16:28.536283 18285 ts_tablet_manager.cc:1939] T a2851c689a474b539f469b2c084c7505 P 45f26718e525426cbe27bf48f96a58cb: Deleting consensus metadata
I20260926 23:16:28.536926 18286 ts_tablet_manager.cc:1918] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:16:28.537096 18286 log.cc:1196] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/a2851c689a474b539f469b2c084c7505
I20260926 23:16:28.538216 18261 tablet_bootstrap.cc:492] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: No bootstrap required, opened a new log
I20260926 23:16:28.538394 18261 ts_tablet_manager.cc:1397] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20260926 23:16:28.539669 17741 catalog_manager.cc:4841] TS 45f26718e525426cbe27bf48f96a58cb (127.16.62.66:46235): tablet a2851c689a474b539f469b2c084c7505 (table test-table [id=da127ed1fac34eac9f4983815fafcf13]) successfully deleted
I20260926 23:16:28.540026 18286 ts_tablet_manager.cc:1939] T a2851c689a474b539f469b2c084c7505 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Deleting consensus metadata
I20260926 23:16:28.540225 18263 tablet_bootstrap.cc:492] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: No bootstrap required, opened a new log
I20260926 23:16:28.540344 18261 raft_consensus.cc:348] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.540506 18263 ts_tablet_manager.cc:1397] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260926 23:16:28.540612 18261 raft_consensus.cc:374] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:28.540771 18261 raft_consensus.cc:729] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f26718e525426cbe27bf48f96a58cb, State: Initialized, Role: FOLLOWER
I20260926 23:16:28.541033 18261 consensus_queue.cc:260] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.541466 18263 raft_consensus.cc:348] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.541682 18263 raft_consensus.cc:374] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:28.541752 18263 raft_consensus.cc:729] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223677f3c68e4a7bb8668f802a45d248, State: Initialized, Role: FOLLOWER
I20260926 23:16:28.541985 18263 consensus_queue.cc:260] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.542083 17741 catalog_manager.cc:4841] TS 3f04a4f42b1b4f0dad87229b22a8a3b7 (127.16.62.65:36453): tablet a2851c689a474b539f469b2c084c7505 (table test-table [id=da127ed1fac34eac9f4983815fafcf13]) successfully deleted
I20260926 23:16:28.542444 18263 ts_tablet_manager.cc:1428] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 23:16:28.543390 18261 ts_tablet_manager.cc:1428] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260926 23:16:28.546345 18262 tablet_bootstrap.cc:492] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: No bootstrap required, opened a new log
I20260926 23:16:28.546495 18262 ts_tablet_manager.cc:1397] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.013s
I20260926 23:16:28.547403 18262 raft_consensus.cc:348] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.547649 18262 raft_consensus.cc:374] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:28.547747 18262 raft_consensus.cc:729] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f04a4f42b1b4f0dad87229b22a8a3b7, State: Initialized, Role: FOLLOWER
I20260926 23:16:28.547974 18262 consensus_queue.cc:260] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.548555 18262 ts_tablet_manager.cc:1428] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 23:16:28.698719 18267 raft_consensus.cc:482] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:28.698956 18267 raft_consensus.cc:504] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.701109 18267 leader_election.cc:288] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f04a4f42b1b4f0dad87229b22a8a3b7 (127.16.62.65:36453), 223677f3c68e4a7bb8668f802a45d248 (127.16.62.67:33961)
I20260926 23:16:28.709465 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.701588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:38027 (local address 127.16.62.67:33961)
0926 23:16:28.701834 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.701843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.702519 (+ 676us) server_negotiation.cc:407] Connection header received
0926 23:16:28.702854 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.702861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.702974 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.703088 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:28.703693 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.704731 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.706985 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.708363 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.708559 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.708659 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.708980 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.709210 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 23:16:28.709892 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.702030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:40353 (local address 127.16.62.65:36453)
0926 23:16:28.702237 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.702244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.703314 (+ 1070us) server_negotiation.cc:407] Connection header received
0926 23:16:28.703480 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.703488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.703614 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.703716 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:28.704316 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.705639 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.707766 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.708700 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.708857 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.708963 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.709427 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.709590 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:16:28.709967 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.701603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:36453 (local address 127.16.62.66:40353)
0926 23:16:28.703201 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:28.703217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:28.703342 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:28.703722 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:28.703726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:28.703756 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:28.704110 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.704121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.705794 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.705799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.707622 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.707639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.708963 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.708971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.709239 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.709284 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 23:16:28.709400 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 23:16:28.709559 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"thread_start_us":672,"threads_started":1}
I20260926 23:16:28.710635 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.701276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:33961 (local address 127.16.62.66:38027)
0926 23:16:28.702395 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:28.702420 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:28.702663 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:28.703162 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:28.703167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:28.703202 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:28.703548 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.703557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.704889 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.704894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.706797 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.706811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.708574 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.708581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.708756 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.708802 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 23:16:28.709902 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0926 23:16:28.710312 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":1659,"threads_started":1}
I20260926 23:16:28.710987 17915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a9300ed78f4a0ab87ae13cf4b48b83" candidate_uuid: "45f26718e525426cbe27bf48f96a58cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" is_pre_election: true
I20260926 23:16:28.711263 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a9300ed78f4a0ab87ae13cf4b48b83" candidate_uuid: "45f26718e525426cbe27bf48f96a58cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223677f3c68e4a7bb8668f802a45d248" is_pre_election: true
I20260926 23:16:28.711524 18196 raft_consensus.cc:2454] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f26718e525426cbe27bf48f96a58cb in term 0.
I20260926 23:16:28.711510 17915 raft_consensus.cc:2454] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f26718e525426cbe27bf48f96a58cb in term 0.
I20260926 23:16:28.711990 17986 leader_election.cc:302] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223677f3c68e4a7bb8668f802a45d248, 45f26718e525426cbe27bf48f96a58cb; no voters:
I20260926 23:16:28.712450 18267 raft_consensus.cc:2781] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:28.712605 18267 raft_consensus.cc:482] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:28.712719 18267 raft_consensus.cc:3037] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:28.716213 18267 raft_consensus.cc:504] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.717221 18267 leader_election.cc:288] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [CANDIDATE]: Term 1 election: Requested vote from peers 3f04a4f42b1b4f0dad87229b22a8a3b7 (127.16.62.65:36453), 223677f3c68e4a7bb8668f802a45d248 (127.16.62.67:33961)
I20260926 23:16:28.717381 17915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a9300ed78f4a0ab87ae13cf4b48b83" candidate_uuid: "45f26718e525426cbe27bf48f96a58cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7"
I20260926 23:16:28.717592 17915 raft_consensus.cc:3037] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:28.717533 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a9300ed78f4a0ab87ae13cf4b48b83" candidate_uuid: "45f26718e525426cbe27bf48f96a58cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223677f3c68e4a7bb8668f802a45d248"
I20260926 23:16:28.717741 18196 raft_consensus.cc:3037] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:28.720748 18196 raft_consensus.cc:2454] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f26718e525426cbe27bf48f96a58cb in term 1.
I20260926 23:16:28.721114 17986 leader_election.cc:302] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223677f3c68e4a7bb8668f802a45d248, 45f26718e525426cbe27bf48f96a58cb; no voters:
I20260926 23:16:28.721333 17915 raft_consensus.cc:2454] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f26718e525426cbe27bf48f96a58cb in term 1.
I20260926 23:16:28.721418 18267 raft_consensus.cc:2781] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:28.721892 18267 raft_consensus.cc:686] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 1 LEADER]: Becoming Leader. State: Replica: 45f26718e525426cbe27bf48f96a58cb, State: Running, Role: LEADER
I20260926 23:16:28.722312 18267 consensus_queue.cc:237] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.725992 17756 catalog_manager.cc:5495] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 45f26718e525426cbe27bf48f96a58cb (127.16.62.66). New cstate: current_term: 1 leader_uuid: "45f26718e525426cbe27bf48f96a58cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } health_report { overall_health: UNKNOWN } } }
Sep 26 23:16:28 dist-test-slave-vcv1 krb5kdc[17712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464586, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:28.790915 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.774351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.16.62.126:44059)
0926 23:16:28.778079 (+ 3728us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:28.778083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:28.778100 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:16:28.778167 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:28.778172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:28.778249 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:28.778357 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:28.779522 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.780367 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.781530 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.781957 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.782094 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.785188 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:28.785192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:28.785195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:28.785220 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:28.787711 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:28.788467 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:28.788473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:28.788479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:28.788598 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:28.789131 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:28.789138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:28.789141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:28.789421 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:28.789595 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:28.790050 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 23:16:28.790417 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":174,"server-negotiator.queue_time_us":3637,"thread_start_us":3541,"threads_started":1}
I20260926 23:16:28.791075 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:28.774343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44059 (local address 127.0.0.1:60548)
0926 23:16:28.774937 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:28.774957 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:28.775077 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:28.778379 (+ 3302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:28.778383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:28.779120 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:28.779375 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.779385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.780483 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.780490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.781398 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:28.781409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:28.782114 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:28.782120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:28.782276 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:28.782919 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:28.782936 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:28.785010 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:28.787828 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:28.787841 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:28.787849 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:28.788360 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:28.788706 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:28.788710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:28.788713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:28.788996 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:28.789559 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:28.789566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:28.789780 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 23:16:28.789964 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 23:16:28.790287 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":322,"threads_started":1}
I20260926 23:16:28.797199 17756 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:60548:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 23:16:28.798686 17756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:28.805881 17894 tablet_service.cc:1467] Processing CreateTablet for tablet e3cb9c03b1204cc8b24226fada81d8c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=293e591b97df452889396e53636f9261]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:16:28.806234 18033 tablet_service.cc:1467] Processing CreateTablet for tablet e3cb9c03b1204cc8b24226fada81d8c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=293e591b97df452889396e53636f9261]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:16:28.806649 17894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3cb9c03b1204cc8b24226fada81d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:28.806950 18033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3cb9c03b1204cc8b24226fada81d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:28.814548 18172 tablet_service.cc:1467] Processing CreateTablet for tablet e3cb9c03b1204cc8b24226fada81d8c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=293e591b97df452889396e53636f9261]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:16:28.815429 18172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3cb9c03b1204cc8b24226fada81d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:28.817380 18262 tablet_bootstrap.cc:492] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Bootstrap starting.
I20260926 23:16:28.819787 18261 tablet_bootstrap.cc:492] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: Bootstrap starting.
I20260926 23:16:28.820348 18262 tablet_bootstrap.cc:654] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:28.822185 18261 tablet_bootstrap.cc:654] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:28.826304 18263 tablet_bootstrap.cc:492] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: Bootstrap starting.
I20260926 23:16:28.828163 18263 tablet_bootstrap.cc:654] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:28.834352 18262 tablet_bootstrap.cc:492] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: No bootstrap required, opened a new log
I20260926 23:16:28.834702 18262 ts_tablet_manager.cc:1397] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260926 23:16:28.835826 18262 raft_consensus.cc:348] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.836248 18262 raft_consensus.cc:374] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:28.836359 18262 raft_consensus.cc:729] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f04a4f42b1b4f0dad87229b22a8a3b7, State: Initialized, Role: FOLLOWER
I20260926 23:16:28.836680 18262 consensus_queue.cc:260] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.837280 18280 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:28.837832 18280 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:28.839521 18263 tablet_bootstrap.cc:492] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: No bootstrap required, opened a new log
I20260926 23:16:28.839608 18262 ts_tablet_manager.cc:1428] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260926 23:16:28.839694 18263 ts_tablet_manager.cc:1397] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260926 23:16:28.840355 18263 raft_consensus.cc:348] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.840528 18263 raft_consensus.cc:374] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:28.840618 18263 raft_consensus.cc:729] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223677f3c68e4a7bb8668f802a45d248, State: Initialized, Role: FOLLOWER
I20260926 23:16:28.840801 18263 consensus_queue.cc:260] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.841238 18263 ts_tablet_manager.cc:1428] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 23:16:28.841908 18269 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:28.842238 18269 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:28.843633 18261 tablet_bootstrap.cc:492] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: No bootstrap required, opened a new log
I20260926 23:16:28.843849 18261 ts_tablet_manager.cc:1397] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.008s
I20260926 23:16:28.844619 18261 raft_consensus.cc:348] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.844887 18261 raft_consensus.cc:374] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:28.844964 18261 raft_consensus.cc:729] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f26718e525426cbe27bf48f96a58cb, State: Initialized, Role: FOLLOWER
I20260926 23:16:28.845264 18261 consensus_queue.cc:260] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.845721 18267 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:28.845988 18267 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:28.846912 18261 ts_tablet_manager.cc:1428] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 23:16:28.982017 18280 raft_consensus.cc:482] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:28.982265 18280 raft_consensus.cc:504] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.982952 18280 leader_election.cc:288] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f26718e525426cbe27bf48f96a58cb (127.16.62.66:46235), 223677f3c68e4a7bb8668f802a45d248 (127.16.62.67:33961)
I20260926 23:16:28.983484 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cb9c03b1204cc8b24226fada81d8c5" candidate_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223677f3c68e4a7bb8668f802a45d248" is_pre_election: true
I20260926 23:16:28.983515 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cb9c03b1204cc8b24226fada81d8c5" candidate_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f26718e525426cbe27bf48f96a58cb" is_pre_election: true
I20260926 23:16:28.983728 18058 raft_consensus.cc:2454] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f04a4f42b1b4f0dad87229b22a8a3b7 in term 0.
I20260926 23:16:28.983750 18196 raft_consensus.cc:2454] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f04a4f42b1b4f0dad87229b22a8a3b7 in term 0.
I20260926 23:16:28.984167 17844 leader_election.cc:302] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f04a4f42b1b4f0dad87229b22a8a3b7, 45f26718e525426cbe27bf48f96a58cb; no voters:
I20260926 23:16:28.984429 18280 raft_consensus.cc:2781] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:28.984566 18280 raft_consensus.cc:482] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:28.984694 18280 raft_consensus.cc:3037] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:28.986755 18280 raft_consensus.cc:504] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.987296 18280 leader_election.cc:288] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [CANDIDATE]: Term 1 election: Requested vote from peers 45f26718e525426cbe27bf48f96a58cb (127.16.62.66:46235), 223677f3c68e4a7bb8668f802a45d248 (127.16.62.67:33961)
I20260926 23:16:28.987732 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cb9c03b1204cc8b24226fada81d8c5" candidate_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223677f3c68e4a7bb8668f802a45d248"
I20260926 23:16:28.987779 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3cb9c03b1204cc8b24226fada81d8c5" candidate_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f26718e525426cbe27bf48f96a58cb"
I20260926 23:16:28.987957 18196 raft_consensus.cc:3037] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:28.987988 18058 raft_consensus.cc:3037] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:28.990809 18196 raft_consensus.cc:2454] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f04a4f42b1b4f0dad87229b22a8a3b7 in term 1.
I20260926 23:16:28.990824 18058 raft_consensus.cc:2454] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f04a4f42b1b4f0dad87229b22a8a3b7 in term 1.
I20260926 23:16:28.991225 17846 leader_election.cc:302] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223677f3c68e4a7bb8668f802a45d248, 3f04a4f42b1b4f0dad87229b22a8a3b7; no voters:
I20260926 23:16:28.991494 18280 raft_consensus.cc:2781] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:28.991945 18280 raft_consensus.cc:686] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 LEADER]: Becoming Leader. State: Replica: 3f04a4f42b1b4f0dad87229b22a8a3b7, State: Running, Role: LEADER
I20260926 23:16:28.992486 18280 consensus_queue.cc:237] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } }
I20260926 23:16:28.994269 18305 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: TxnStatusTablet state changed. Reason: New leader 3f04a4f42b1b4f0dad87229b22a8a3b7. Latest consensus state: current_term: 1 leader_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:28.994613 18305 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:16:28.995507 18307 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 23:16:28.995760 18307 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 23:16:28.995882 17756 catalog_manager.cc:5495] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f04a4f42b1b4f0dad87229b22a8a3b7 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } health_report { overall_health: UNKNOWN } } }
I20260926 23:16:29.022044 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.014771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.16.62.65:36453)
0926 23:16:29.015049 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:29.015056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:29.015075 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:16:29.015151 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:29.015160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:29.015236 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:29.015337 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:29.015798 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.016980 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.019419 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.020088 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.020282 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.021253 (+ 971us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:29.021359 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:29.021429 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:29.021616 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 23:16:29.021681 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 23:16:29.022351 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.014021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:36453 (local address 127.0.0.1:40544)
0926 23:16:29.014249 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:29.014265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:29.014400 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:29.015385 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:29.015390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:29.015404 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:16:29.015634 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:29.015645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.017164 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.017169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:29.019212 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:29.019227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.020367 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.020374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:29.020868 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.020949 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:16:29.021460 (+ 511us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:16:29.021492 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:16:29.021593 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 23:16:29.021808 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 23:16:29.028115 18196 raft_consensus.cc:1261] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248 [term 1 FOLLOWER]: Refusing update from remote peer 3f04a4f42b1b4f0dad87229b22a8a3b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:29.028550 18058 raft_consensus.cc:1261] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb [term 1 FOLLOWER]: Refusing update from remote peer 3f04a4f42b1b4f0dad87229b22a8a3b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:29.028905 18305 consensus_queue.cc:1035] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:29.029420 18280 consensus_queue.cc:1035] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:29.050810 18269 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: TxnStatusTablet state changed. Reason: New leader 3f04a4f42b1b4f0dad87229b22a8a3b7. Latest consensus state: current_term: 1 leader_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:29.051153 18269 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:29.064689 18267 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: TxnStatusTablet state changed. Reason: New leader 3f04a4f42b1b4f0dad87229b22a8a3b7. Latest consensus state: current_term: 1 leader_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:29.065620 18305 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:29.065869 18305 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:16:29.066363 18310 mvcc.cc:205] Tried to move back new op lower bound from 7333742956652974080 to 7333742956515348480. Current Snapshot: MvccSnapshot[applied={T|T < 7333742956652974080}]
I20260926 23:16:29.066613 18309 mvcc.cc:205] Tried to move back new op lower bound from 7333742956652974080 to 7333742956515348480. Current Snapshot: MvccSnapshot[applied={T|T < 7333742956652974080 or (T in {7333742956652974080})}]
I20260926 23:16:29.068255 18267 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:29.072018 18269 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:29.072268 18269 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 223677f3c68e4a7bb8668f802a45d248: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:29.079166 18280 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:29.080024 18280 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 3f04a4f42b1b4f0dad87229b22a8a3b7: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:16:29.084314 18311 mvcc.cc:205] Tried to move back new op lower bound from 7333742956652974080 to 7333742956515348480. Current Snapshot: MvccSnapshot[applied={T|T < 7333742956652974080}]
I20260926 23:16:29.087582 18267 tablet_replica.cc:439] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 } } peers { permanent_uuid: "45f26718e525426cbe27bf48f96a58cb" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 46235 } } peers { permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 } } }
I20260926 23:16:29.087870 18267 tablet_replica.cc:442] T e3cb9c03b1204cc8b24226fada81d8c5 P 45f26718e525426cbe27bf48f96a58cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:29.101115 18267 consensus_queue.cc:1035] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "223677f3c68e4a7bb8668f802a45d248" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:29.104013 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.086018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.16.62.66:46235)
0926 23:16:29.096526 (+ 10508us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:29.096533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:29.096553 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:16:29.096655 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:29.096660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:29.096776 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:29.096955 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:29.097422 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.098311 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.100024 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.100640 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.100770 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.101450 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:29.102157 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:29.102285 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:29.103351 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0926 23:16:29.103455 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10375,"thread_start_us":10260,"threads_started":1}
I20260926 23:16:29.123728 18293 consensus_queue.cc:1035] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f04a4f42b1b4f0dad87229b22a8a3b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:29.133455 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.125995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:38231 (local address 127.16.62.65:36453)
0926 23:16:29.126229 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:29.126238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:29.126901 (+ 663us) server_negotiation.cc:407] Connection header received
0926 23:16:29.127261 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:29.127268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:29.127375 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:29.127445 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:29.128970 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.129797 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.131795 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.132296 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.132385 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.132853 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:29.132976 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:29.133037 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:29.133177 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 23:16:29.133231 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:16:29.133643 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.125810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:36453 (local address 127.16.62.66:38231)
0926 23:16:29.126798 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:29.126822 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:29.127098 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:29.127770 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:29.127776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:29.127791 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:16:29.128116 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:29.128129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.129934 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.129940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:29.131644 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:29.131659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.132454 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.132461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:29.132634 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.132706 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:16:29.133063 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:16:29.133074 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:16:29.133153 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 23:16:29.133286 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":601,"threads_started":1}
I20260926 23:16:29.173164 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.166314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:54439 (local address 127.16.62.66:46235)
0926 23:16:29.166476 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:29.166498 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:29.167361 (+ 863us) server_negotiation.cc:407] Connection header received
0926 23:16:29.167650 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:29.167655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:29.167732 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:29.167797 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:29.168672 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.169509 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.171210 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.171803 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.171931 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.172548 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:29.172638 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:29.172714 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:29.172862 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 23:16:29.172921 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:16:29.173420 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.166283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:46235 (local address 127.16.62.65:54439)
0926 23:16:29.167264 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:29.167280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:29.167487 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:29.167977 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:29.167980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:29.167990 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:16:29.168483 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:29.168501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.169616 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.169623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:29.171050 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:29.171061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.172100 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.172106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:29.172322 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.172387 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:16:29.172742 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:16:29.172753 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:16:29.172840 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 23:16:29.172975 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":561,"threads_started":1}
I20260926 23:16:29.219370 17755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:29.219625 17755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:29.223116 17755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e9829ad0470494facef0e31cc7abb14: Sending DeleteTablet for 3 replicas of tablet f0a9300ed78f4a0ab87ae13cf4b48b83
I20260926 23:16:29.223896 18033 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a9300ed78f4a0ab87ae13cf4b48b83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:29 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:43946
I20260926 23:16:29.224596 18172 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a9300ed78f4a0ab87ae13cf4b48b83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:29 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:33954
I20260926 23:16:29.229385 17894 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a9300ed78f4a0ab87ae13cf4b48b83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:29 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:40534
I20260926 23:16:29.232553 18332 tablet_replica.cc:330] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: stopping tablet replica
I20260926 23:16:29.232885 18332 raft_consensus.cc:2229] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:29.233255 18332 raft_consensus.cc:2258] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:29.234715 18332 ts_tablet_manager.cc:1905] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:29.251094 18334 tablet_replica.cc:330] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: stopping tablet replica
I20260926 23:16:29.251463 18334 raft_consensus.cc:2229] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:29.252019 18331 tablet_replica.cc:330] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: stopping tablet replica
I20260926 23:16:29.252403 18334 raft_consensus.cc:2258] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:29.252667 18331 raft_consensus.cc:2229] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:16:29.253402 18331 raft_consensus.cc:2258] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:29.254498 18334 ts_tablet_manager.cc:1905] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:29.256107 18332 ts_tablet_manager.cc:1918] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:16:29.256359 18332 log.cc:1196] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/f0a9300ed78f4a0ab87ae13cf4b48b83
I20260926 23:16:29.256896 18332 ts_tablet_manager.cc:1939] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 223677f3c68e4a7bb8668f802a45d248: Deleting consensus metadata
I20260926 23:16:29.256932 18331 ts_tablet_manager.cc:1905] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:29.258315 17743 catalog_manager.cc:4841] TS 223677f3c68e4a7bb8668f802a45d248 (127.16.62.67:33961): tablet f0a9300ed78f4a0ab87ae13cf4b48b83 (table test-table [id=4231447c60834357aa8a3c379e45244d]) successfully deleted
I20260926 23:16:29.272058 18331 ts_tablet_manager.cc:1918] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:16:29.272253 18331 log.cc:1196] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/f0a9300ed78f4a0ab87ae13cf4b48b83
I20260926 23:16:29.272778 18331 ts_tablet_manager.cc:1939] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 45f26718e525426cbe27bf48f96a58cb: Deleting consensus metadata
I20260926 23:16:29.274076 17741 catalog_manager.cc:4841] TS 45f26718e525426cbe27bf48f96a58cb (127.16.62.66:46235): tablet f0a9300ed78f4a0ab87ae13cf4b48b83 (table test-table [id=4231447c60834357aa8a3c379e45244d]) successfully deleted
I20260926 23:16:29.276361 18334 ts_tablet_manager.cc:1918] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:16:29.276573 18334 log.cc:1196] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/f0a9300ed78f4a0ab87ae13cf4b48b83
I20260926 23:16:29.277259 18334 ts_tablet_manager.cc:1939] T f0a9300ed78f4a0ab87ae13cf4b48b83 P 3f04a4f42b1b4f0dad87229b22a8a3b7: Deleting consensus metadata
I20260926 23:16:29.278484 17741 catalog_manager.cc:4841] TS 3f04a4f42b1b4f0dad87229b22a8a3b7 (127.16.62.65:36453): tablet f0a9300ed78f4a0ab87ae13cf4b48b83 (table test-table [id=4231447c60834357aa8a3c379e45244d]) successfully deleted
Sep 26 23:16:29 dist-test-slave-vcv1 krb5kdc[17712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:29.292933 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.239753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.16.62.65:36453)
0926 23:16:29.239933 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:29.239940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:29.268406 (+ 28466us) server_negotiation.cc:407] Connection header received
0926 23:16:29.268662 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:29.268669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:29.268768 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:29.268873 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:29.275952 (+ 7079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.277202 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.279160 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.279841 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.279986 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.285066 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:29.285083 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:29.285105 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:29.285163 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:29.289342 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.290188 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.290198 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.290202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.290355 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.290789 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.290795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.290798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.291257 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:29.291546 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:29.292110 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 23:16:29.292390 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 23:16:29.293697 17959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40550
I20260926 23:16:29.319350 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.296790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.16.62.126:44059)
0926 23:16:29.303423 (+ 6633us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:29.303430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:29.303690 (+ 260us) server_negotiation.cc:407] Connection header received
0926 23:16:29.303855 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:29.303860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:29.303941 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:29.304056 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:29.305932 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.307218 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.309099 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.309741 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.309927 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.312931 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:29.312938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:29.312943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:29.312991 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:29.316514 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.317291 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.317298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.317303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.317450 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.317846 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.317851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.317854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.318179 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:29.318421 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:29.318754 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 23:16:29.318972 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6497,"thread_start_us":6393,"threads_started":1}
W20260926 23:16:29.319950 17755 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60554
I20260926 23:16:29.322964 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17828
I20260926 23:16:29.341995 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.324839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.16.62.65:36453)
0926 23:16:29.325010 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:29.325016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:29.325035 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:16:29.325267 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:29.325274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:29.325358 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:29.325429 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:29.327061 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.328256 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.330060 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.330802 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.330926 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.333890 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:29.333897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:29.333902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:29.333990 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:29.338118 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.338962 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.338967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.338971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.339196 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.339653 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.339658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.339660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.340122 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:29.340860 (+ 738us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:29.340988 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 23:16:29.341282 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 23:16:29.342780 17959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40560
W20260926 23:16:29.343477 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17828: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:16:29.343835 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 17828
I20260926 23:16:29.376791 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17968
Sep 26 23:16:29 dist-test-slave-vcv1 krb5kdc[17712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:16:29.394249 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.377653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.16.62.66:46235)
0926 23:16:29.377876 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:29.377882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:29.377902 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:16:29.378104 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:29.378115 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:29.378196 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:29.378316 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:29.380011 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.381097 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.382677 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.383386 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.383547 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.387911 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:29.387924 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:29.387938 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:29.387970 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:29.391202 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.391904 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.391920 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.391923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.392088 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.392580 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.392586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.392590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.392983 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:29.393329 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:29.393625 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 23:16:29.393867 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 23:16:29.394982 18098 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43970
W20260926 23:16:29.395716 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17968: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:16:29.395887 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 17968
I20260926 23:16:29.413993 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 18107
Sep 26 23:16:29 dist-test-slave-vcv1 krb5kdc[17712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:16:29.428561 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:29.414778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.16.62.67:33961)
0926 23:16:29.415283 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:29.415287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:29.415303 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:16:29.415352 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:29.415355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:29.415412 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:29.415499 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:29.416719 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.417386 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.418530 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:29.418982 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:29.419087 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:29.423341 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:29.423355 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:29.423370 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:29.423408 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:29.426002 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.426659 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.426666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.426668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.426783 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:29.427116 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:29.427119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:29.427121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:29.427442 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:29.427642 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:29.427996 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 23:16:29.428211 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":305,"threads_started":1}
W20260926 23:16:29.429332 18237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33966
W20260926 23:16:29.429986 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 18107: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:16:29.430148 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 18107
I20260926 23:16:29.446491 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 17725
I20260926 23:16:29.447576 17797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:29.560597 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 17725
2026-09-26T23:16:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4790 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:16:29 dist-test-slave-vcv1 krb5kdc[18344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:16:29 dist-test-slave-vcv1 krb5kdc[18344](info): set up 2 sockets
Sep 26 23:16:29 dist-test-slave-vcv1 krb5kdc[18344](info): commencing operation
krb5kdc: starting...
W20260926 23:16:30.629027 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:16:30 dist-test-slave-vcv1 krb5kdc[18344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:16:30.824498 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:36325
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:31.156678 18357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:31.156970 18357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:31.157166 18357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:31.164417 18357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:31.164520 18357 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 23:16:31.164613 18357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:31.164718 18357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:31.164811 18357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:31.164924 18357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:31.174813 18357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:36325
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:31.176123 18357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:31.177500 18357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:31.189581 18362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:31.189633 18365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:31.189667 18363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:31.190790 18357 server_base.cc:1034] running on GCE node
I20260926 23:16:31.191587 18357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:31.192924 18357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:31.194185 18357 hybrid_clock.cc:648] HybridClock initialized: now 1790464591194113 us; error 44 us; skew 500 ppm
I20260926 23:16:31.194540 18357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:31 dist-test-slave-vcv1 krb5kdc[18344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:31.197657 18357 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:16:31.199002 18357 webserver.cc:460] Webserver started at http://127.16.62.126:35241/ using document root <none> and password file <none>
I20260926 23:16:31.199702 18357 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:31.199833 18357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:31.200191 18357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:31.203038 18357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "5b0f894e568f4e3fb4a8e699e734b8bc"
format_stamp: "Formatted at 2026-09-26 23:16:31 on dist-test-slave-vcv1"
server_key: "5416d8929f248135117de7232ccf9933"
server_key_iv: "b264c06faaa399644172ea30479048d7"
server_key_version: "encryptionkey@0"
I20260926 23:16:31.203763 18357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "5b0f894e568f4e3fb4a8e699e734b8bc"
format_stamp: "Formatted at 2026-09-26 23:16:31 on dist-test-slave-vcv1"
server_key: "5416d8929f248135117de7232ccf9933"
server_key_iv: "b264c06faaa399644172ea30479048d7"
server_key_version: "encryptionkey@0"
I20260926 23:16:31.208703 18357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 23:16:31.211993 18372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:31.213235 18357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:31.213403 18357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "5b0f894e568f4e3fb4a8e699e734b8bc"
format_stamp: "Formatted at 2026-09-26 23:16:31 on dist-test-slave-vcv1"
server_key: "5416d8929f248135117de7232ccf9933"
server_key_iv: "b264c06faaa399644172ea30479048d7"
server_key_version: "encryptionkey@0"
I20260926 23:16:31.213609 18357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:31.231174 18357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:31.234979 18357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:31.235244 18357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:31.256656 18357 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:36325
I20260926 23:16:31.256692 18433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:36325 every 8 connection(s)
I20260926 23:16:31.257905 18357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:31.261611 18434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:31.263108 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 18357
I20260926 23:16:31.263343 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:31.264285 16633 external_mini_cluster.cc:1428] Setting key 7e3cf2b8b50eab1f3b57cd0906e5b319
I20260926 23:16:31.276576 18434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc: Bootstrap starting.
Sep 26 23:16:31 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:31.280021 18434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:31.281260 18434 log.cc:824] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:31.283946 18434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc: No bootstrap required, opened a new log
I20260926 23:16:31.286381 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:31.266375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.16.62.126:36325)
0926 23:16:31.267140 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:31.267151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:31.267191 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:16:31.268764 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:31.268785 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:31.269142 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:31.269448 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:31.271203 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.272495 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.274280 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.274960 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.275093 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:31.279525 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:31.279532 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:31.279546 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:31.279582 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:31.282571 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:31.283213 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:31.283220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:31.283227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:31.283391 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:31.283749 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:31.283756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:31.283762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:31.284345 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:31.284645 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:31.285044 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 23:16:31.285435 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":367,"threads_started":1}
I20260926 23:16:31.291417 18434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f894e568f4e3fb4a8e699e734b8bc" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36325 } }
I20260926 23:16:31.291767 18434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:31.291880 18434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0f894e568f4e3fb4a8e699e734b8bc, State: Initialized, Role: FOLLOWER
I20260926 23:16:31.292486 18434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f894e568f4e3fb4a8e699e734b8bc" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36325 } }
I20260926 23:16:31.292728 18434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:31.292862 18434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:31.293025 18434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:31.295519 18434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f894e568f4e3fb4a8e699e734b8bc" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36325 } }
I20260926 23:16:31.296010 18434 leader_election.cc:302] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b0f894e568f4e3fb4a8e699e734b8bc; no voters:
I20260926 23:16:31.296540 18434 leader_election.cc:288] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:31.296676 18439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:31.297374 18439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [term 1 LEADER]: Becoming Leader. State: Replica: 5b0f894e568f4e3fb4a8e699e734b8bc, State: Running, Role: LEADER
I20260926 23:16:31.297894 18439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f894e568f4e3fb4a8e699e734b8bc" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36325 } }
I20260926 23:16:31.298509 18434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:31.300645 18440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b0f894e568f4e3fb4a8e699e734b8bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f894e568f4e3fb4a8e699e734b8bc" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36325 } } }
I20260926 23:16:31.300654 18441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b0f894e568f4e3fb4a8e699e734b8bc. Latest consensus state: current_term: 1 leader_uuid: "5b0f894e568f4e3fb4a8e699e734b8bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0f894e568f4e3fb4a8e699e734b8bc" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36325 } } }
I20260926 23:16:31.301091 18440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:31.301091 18441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:31.309334 18445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:31.313586 18445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:31.321292 18445 catalog_manager.cc:1347] Generated new cluster ID: aa1226cae3784513b02f3e26276086e4
I20260926 23:16:31.321416 18445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:31.333438 18445 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:31.334724 18445 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:31.341480 18445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc: Generated new TSK 0
I20260926 23:16:31.342276 18445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:16:31.397370 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36325
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:31.716337 18460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:31.716686 18460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:31.716874 18460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:31.724063 18460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:31.724215 18460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:31.724306 18460 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 23:16:31.724362 18460 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:16:31.724426 18460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:16:31.733769 18460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.62.126:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:31.734910 18460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:31.736188 18460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:31.750118 18465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:31.750150 18466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:31.750078 18468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:31.751228 18460 server_base.cc:1034] running on GCE node
I20260926 23:16:31.751832 18460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:31.752787 18460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:31.754005 18460 hybrid_clock.cc:648] HybridClock initialized: now 1790464591753957 us; error 60 us; skew 500 ppm
I20260926 23:16:31.754310 18460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:31 dist-test-slave-vcv1 krb5kdc[18344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:31.757366 18460 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:16:31.758632 18460 webserver.cc:460] Webserver started at http://127.16.62.65:45081/ using document root <none> and password file <none>
I20260926 23:16:31.759315 18460 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:31.759439 18460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:31.759759 18460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:31.762594 18460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "056163913b27480b830b03784577d190"
format_stamp: "Formatted at 2026-09-26 23:16:31 on dist-test-slave-vcv1"
server_key: "32389af47c13e81efd172dc56ddbcb93"
server_key_iv: "3cc0eb39a659186cc25c299d691ae227"
server_key_version: "encryptionkey@0"
I20260926 23:16:31.763391 18460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "056163913b27480b830b03784577d190"
format_stamp: "Formatted at 2026-09-26 23:16:31 on dist-test-slave-vcv1"
server_key: "32389af47c13e81efd172dc56ddbcb93"
server_key_iv: "3cc0eb39a659186cc25c299d691ae227"
server_key_version: "encryptionkey@0"
I20260926 23:16:31.768319 18460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 23:16:31.771580 18475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:31.772835 18460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:31.772993 18460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "056163913b27480b830b03784577d190"
format_stamp: "Formatted at 2026-09-26 23:16:31 on dist-test-slave-vcv1"
server_key: "32389af47c13e81efd172dc56ddbcb93"
server_key_iv: "3cc0eb39a659186cc25c299d691ae227"
server_key_version: "encryptionkey@0"
I20260926 23:16:31.773166 18460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:31.813463 18460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:31.817693 18460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:31.817986 18460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:31.820536 18460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:31.820672 18460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:31.820801 18460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:31.820890 18460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:16:31 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:31.843711 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:31.821286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:38083 (local address 127.16.62.126:36325)
0926 23:16:31.821838 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:31.821844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:31.823053 (+ 1209us) server_negotiation.cc:407] Connection header received
0926 23:16:31.826090 (+ 3037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:31.826096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:31.826202 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:31.826342 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:31.828571 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.829474 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.831789 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.832291 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.832445 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:31.837401 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:31.837409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:31.837415 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:31.837465 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:31.840605 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:31.841496 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:31.841501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:31.841504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:31.841639 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:31.842073 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:31.842078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:31.842081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:31.842338 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:31.842482 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:31.843109 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 23:16:31.843316 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":308,"threads_started":1}
I20260926 23:16:31.844830 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:31.821986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36325 (local address 127.16.62.65:38083)
0926 23:16:31.822845 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:31.822899 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:31.825730 (+ 2831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:31.826586 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:31.826593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:31.827319 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:31.828372 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:31.828401 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.829690 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.829696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:31.831645 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:31.831656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.832546 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.832556 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:31.832923 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:31.833839 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:31.833874 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:31.837171 (+ 3297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:31.840771 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:31.840778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:31.840798 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:31.841377 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:31.841768 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:31.841776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:31.841779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:31.841962 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:31.842457 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:31.842465 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:31.842783 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 23:16:31.843022 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 23:16:31.843501 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":360,"threads_started":1}
I20260926 23:16:31.870229 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:31.855333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36325 (local address 127.16.62.65:53439)
0926 23:16:31.856016 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:31.856040 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:31.856215 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:31.856616 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:31.856621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:31.857332 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:31.857788 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:31.857806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.859130 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.859137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:31.860940 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:31.860950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.862061 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.862065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:31.862189 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:31.862813 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:31.862828 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:31.863947 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:31.867490 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:31.867497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:31.867506 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:31.868017 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:31.868483 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:31.868488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:31.868492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:31.868672 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:31.869419 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:31.869422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:31.869566 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 23:16:31.869680 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 23:16:31.869894 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":295,"threads_started":1}
I20260926 23:16:31.870601 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:31.855022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:53439 (local address 127.16.62.126:36325)
0926 23:16:31.855212 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:31.855218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:31.856185 (+ 967us) server_negotiation.cc:407] Connection header received
0926 23:16:31.856389 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:31.856394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:31.856475 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:31.856592 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:31.857945 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.858983 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.861146 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:31.861719 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:31.861852 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:31.864236 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:31.864241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:31.864244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:31.864271 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:31.867324 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:31.868209 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:31.868215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:31.868218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:31.868372 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:31.868835 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:31.868842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:31.868849 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:31.869294 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:31.869479 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:31.869821 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 23:16:31.870106 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 23:16:31.874452 18460 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:36969
I20260926 23:16:31.874485 18595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:36969 every 8 connection(s)
I20260926 23:16:31.875614 18460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:16:31.877667 18596 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36325
I20260926 23:16:31.877844 18596 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:31.878381 18596 heartbeater.cc:511] Master 127.16.62.126:36325 requested a full tablet report, sending...
I20260926 23:16:31.880106 18389 ts_manager.cc:194] Registered new tserver with Master: 056163913b27480b830b03784577d190 (127.16.62.65:36969)
I20260926 23:16:31.881546 18389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:38083
I20260926 23:16:31.885447 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 18460
I20260926 23:16:31.885668 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:16:31.886276 16633 external_mini_cluster.cc:1428] Setting key 1812b0de5639c234d73d07ef47f1e1b9
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:16:31.923430 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36325
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:32.233523 18600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:32.233804 18600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:32.233973 18600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:32.241243 18600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:32.241350 18600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:32.241498 18600 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 23:16:32.241561 18600 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:16:32.241675 18600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:16:32.251300 18600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.62.126:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:32.252628 18600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:32.253942 18600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:16:32.267419 18600 server_base.cc:1034] running on GCE node
W20260926 23:16:32.267333 18606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:32.267338 18605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:32.267836 18608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:32.268551 18600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:32.269426 18600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:32.270677 18600 hybrid_clock.cc:648] HybridClock initialized: now 1790464592270620 us; error 51 us; skew 500 ppm
I20260926 23:16:32.271080 18600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:32 dist-test-slave-vcv1 krb5kdc[18344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:32.274186 18600 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:16:32.275543 18600 webserver.cc:460] Webserver started at http://127.16.62.66:44305/ using document root <none> and password file <none>
I20260926 23:16:32.276280 18600 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:32.276420 18600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:32.276731 18600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:32.279635 18600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f"
format_stamp: "Formatted at 2026-09-26 23:16:32 on dist-test-slave-vcv1"
server_key: "7242292d6a41fcfe6c8b99d490cb1d86"
server_key_iv: "4d11bc5aeab4da8291100ae19c98ed6d"
server_key_version: "encryptionkey@0"
I20260926 23:16:32.280432 18600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f"
format_stamp: "Formatted at 2026-09-26 23:16:32 on dist-test-slave-vcv1"
server_key: "7242292d6a41fcfe6c8b99d490cb1d86"
server_key_iv: "4d11bc5aeab4da8291100ae19c98ed6d"
server_key_version: "encryptionkey@0"
I20260926 23:16:32.285401 18600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:16:32.288851 18615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:32.290158 18600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:16:32.290330 18600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f"
format_stamp: "Formatted at 2026-09-26 23:16:32 on dist-test-slave-vcv1"
server_key: "7242292d6a41fcfe6c8b99d490cb1d86"
server_key_iv: "4d11bc5aeab4da8291100ae19c98ed6d"
server_key_version: "encryptionkey@0"
I20260926 23:16:32.290517 18600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:32.303649 18600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:32.307675 18600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:32.307971 18600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:32.310644 18600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:32.310789 18600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:32.310928 18600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:32.311043 18600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:16:32 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:32.334642 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.311209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:34149 (local address 127.16.62.126:36325)
0926 23:16:32.311458 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:32.311465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:32.312884 (+ 1419us) server_negotiation.cc:407] Connection header received
0926 23:16:32.314897 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:32.314903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:32.314991 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:32.315103 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:32.317524 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.318514 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.321194 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.321639 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.321777 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.326663 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:32.326668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:32.326671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:32.326703 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:32.330638 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.331881 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.331890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.331896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.332075 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.332714 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.332719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.332724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.333146 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:32.333314 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:32.334068 (+ 754us) server_negotiation.cc:299] Negotiation successful
0926 23:16:32.334286 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:16:32.335697 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.311825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36325 (local address 127.16.62.66:34149)
0926 23:16:32.312695 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:32.312743 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:32.314598 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:32.315375 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:32.315388 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:32.316216 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:32.317219 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.317251 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.318654 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.318662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.321021 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.321036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.321869 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.321876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.322109 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.322847 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:32.322885 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:32.326420 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:32.330837 (+ 4417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:32.330847 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:32.330873 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:32.331686 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:32.332253 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:32.332261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:32.332267 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:32.332578 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:32.333299 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:32.333307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:32.333672 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 23:16:32.333952 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 23:16:32.334513 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":419,"threads_started":1}
I20260926 23:16:32.356611 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.342006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:32957 (local address 127.16.62.126:36325)
0926 23:16:32.342177 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:32.342181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:32.342691 (+ 510us) server_negotiation.cc:407] Connection header received
0926 23:16:32.342889 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:32.342893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:32.342951 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:32.343048 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:32.343955 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.344960 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.346757 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.347261 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.347412 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.349630 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:32.349640 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:32.349648 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:32.349699 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:32.353866 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.354489 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.354497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.354503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.354681 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.355191 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.355195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.355197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.355588 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:32.355761 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:32.356239 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 23:16:32.356394 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 23:16:32.357124 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.342019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36325 (local address 127.16.62.66:32957)
0926 23:16:32.342584 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:32.342601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:32.342760 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:32.343071 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:32.343075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:32.343541 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:32.343819 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.343828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.345166 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.345174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.346612 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.346623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.347454 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.347461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.347620 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.348269 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:32.348285 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:32.349391 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:32.354006 (+ 4615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:32.354009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:32.354012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:32.354382 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:32.354812 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:32.354820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:32.354829 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:32.355013 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:32.355735 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:32.355740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:32.355952 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 23:16:32.356132 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 23:16:32.356501 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":324,"threads_started":1}
I20260926 23:16:32.364563 18600 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:36023
I20260926 23:16:32.364598 18734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:36023 every 8 connection(s)
I20260926 23:16:32.365754 18600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:16:32.367669 18735 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36325
I20260926 23:16:32.367890 18735 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:32.368530 18735 heartbeater.cc:511] Master 127.16.62.126:36325 requested a full tablet report, sending...
I20260926 23:16:32.369858 18389 ts_manager.cc:194] Registered new tserver with Master: cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66:36023)
I20260926 23:16:32.370167 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 18600
I20260926 23:16:32.370369 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:16:32.370806 16633 external_mini_cluster.cc:1428] Setting key 58680307406bd6d446a1b3febae137ac
I20260926 23:16:32.370889 18389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:34149
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:16:32.411275 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36325
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:32.722950 18739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:32.723217 18739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:32.723313 18739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:32.730537 18739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:32.730641 18739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:32.730731 18739 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 23:16:32.730785 18739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:16:32.730852 18739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:16:32.740763 18739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.62.126:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:32.741936 18739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:32.743149 18739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:32.755890 18744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:32.755822 18745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:32.755920 18747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:32.756533 18739 server_base.cc:1034] running on GCE node
I20260926 23:16:32.757115 18739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:32.757993 18739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:32.759224 18739 hybrid_clock.cc:648] HybridClock initialized: now 1790464592759184 us; error 54 us; skew 500 ppm
I20260926 23:16:32.759526 18739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:32 dist-test-slave-vcv1 krb5kdc[18344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:32.778558 18739 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:16:32.780104 18739 webserver.cc:460] Webserver started at http://127.16.62.67:37003/ using document root <none> and password file <none>
I20260926 23:16:32.780850 18739 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:32.780999 18739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:32.781322 18739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:32.784188 18739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "40689cb6f0814e2cae383be36c175beb"
format_stamp: "Formatted at 2026-09-26 23:16:32 on dist-test-slave-vcv1"
server_key: "8db45967a42357c0518735d2c3a90080"
server_key_iv: "b0400f98d9e1dd518c250fbad0bc3cc3"
server_key_version: "encryptionkey@0"
I20260926 23:16:32.784921 18739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "40689cb6f0814e2cae383be36c175beb"
format_stamp: "Formatted at 2026-09-26 23:16:32 on dist-test-slave-vcv1"
server_key: "8db45967a42357c0518735d2c3a90080"
server_key_iv: "b0400f98d9e1dd518c250fbad0bc3cc3"
server_key_version: "encryptionkey@0"
I20260926 23:16:32.790187 18739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:16:32.793493 18754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:32.794880 18739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:16:32.795045 18739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "40689cb6f0814e2cae383be36c175beb"
format_stamp: "Formatted at 2026-09-26 23:16:32 on dist-test-slave-vcv1"
server_key: "8db45967a42357c0518735d2c3a90080"
server_key_iv: "b0400f98d9e1dd518c250fbad0bc3cc3"
server_key_version: "encryptionkey@0"
I20260926 23:16:32.795240 18739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:32.816021 18739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:32.820396 18739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:32.820681 18739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:32.823334 18739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:32.823463 18739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:32.823616 18739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:32.823745 18739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:16:32 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:32.848382 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.823772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:36577 (local address 127.16.62.126:36325)
0926 23:16:32.823964 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:32.823969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:32.825464 (+ 1495us) server_negotiation.cc:407] Connection header received
0926 23:16:32.827866 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:32.827873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:32.827981 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:32.828134 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:32.830872 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.831909 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.835405 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.835815 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.835911 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.841493 (+ 5582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:32.841501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:32.841507 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:32.841551 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:32.844843 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.845769 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.845774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.845778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.845965 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.846626 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.846631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.846635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.846950 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:32.847114 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:32.847790 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 23:16:32.848008 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:16:32.849375 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.824424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36325 (local address 127.16.62.67:36577)
0926 23:16:32.825282 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:32.825327 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:32.827540 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:32.828475 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:32.828494 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:32.829470 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:32.830640 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.830681 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.832100 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.832108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.835215 (+ 3107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.835233 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.836069 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.836079 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.836453 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.837319 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:32.837348 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:32.841224 (+ 3876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:32.844981 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:32.844990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:32.845008 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:32.845616 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:32.846208 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:32.846215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:32.846221 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:32.846519 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:32.847076 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:32.847083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:32.847411 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 23:16:32.847654 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 23:16:32.848165 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":358,"threads_started":1}
I20260926 23:16:32.875890 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.859538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:55561 (local address 127.16.62.126:36325)
0926 23:16:32.859731 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:32.859738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:32.860834 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 23:16:32.861022 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:32.861026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:32.861092 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:32.861163 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:32.862658 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.863554 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.866138 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.866619 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.866742 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.869808 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:32.869815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:32.869826 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:32.869864 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:32.872900 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.873784 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.873790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.873796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.873947 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.874392 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.874398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.874404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.874754 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:32.874891 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:32.875294 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 23:16:32.875509 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:16:32.875981 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.859637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36325 (local address 127.16.62.67:55561)
0926 23:16:32.860705 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:32.860730 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:32.860899 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:32.861229 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:32.861236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:32.862005 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:32.862441 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.862455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.863719 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.863727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.865973 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.865990 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.866838 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.866847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.867075 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.867976 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:32.868010 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:32.869617 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:32.873073 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:32.873077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:32.873081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:32.873596 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:32.874079 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:32.874083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:32.874087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:32.874230 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:32.874867 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:32.874884 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:32.875053 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 23:16:32.875187 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 23:16:32.875509 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":686,"threads_started":1}
I20260926 23:16:32.883899 18596 heartbeater.cc:503] Master 127.16.62.126:36325 was elected leader, sending a full tablet report...
I20260926 23:16:32.890869 18739 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:33275
I20260926 23:16:32.890902 18873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:33275 every 8 connection(s)
I20260926 23:16:32.892096 18739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:16:32.894294 18874 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36325
I20260926 23:16:32.894515 18874 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:32.895176 18874 heartbeater.cc:511] Master 127.16.62.126:36325 requested a full tablet report, sending...
I20260926 23:16:32.896487 18389 ts_manager.cc:194] Registered new tserver with Master: 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275)
I20260926 23:16:32.897293 18389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:36577
I20260926 23:16:32.898138 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 18739
I20260926 23:16:32.898365 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:16:32.898859 16633 external_mini_cluster.cc:1428] Setting key a79e734d8e097dea7bad1ff8e9832aaa
I20260926 23:16:32.901790 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:16:32 dist-test-slave-vcv1 krb5kdc[18344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:16:32 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:32.944896 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.932431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.16.62.126:36325)
0926 23:16:32.932652 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:32.932658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:32.932827 (+ 169us) server_negotiation.cc:407] Connection header received
0926 23:16:32.932967 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:32.932971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:32.933032 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:32.933113 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:32.934357 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.935037 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.936282 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.936686 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.936780 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.940550 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:32.940554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:32.940558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:32.940587 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:32.942821 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.943373 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.943377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.943379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.943487 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:32.943789 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:32.943792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:32.943794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:32.944029 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:32.944276 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:32.944499 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 23:16:32.944639 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 23:16:32.948874 18389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:16:32.955902 18389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:32.976585 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.967019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.16.62.67:33275)
0926 23:16:32.968696 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:32.968718 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:32.968755 (+ 37us) server_negotiation.cc:407] Connection header received
0926 23:16:32.968851 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:32.968857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:32.969106 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:32.970144 (+ 1038us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:32.970440 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.971994 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.974333 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.975262 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.975401 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.975638 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:32.976067 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 23:16:32.976256 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1507,"thread_start_us":1344,"threads_started":1}
I20260926 23:16:32.976643 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.966843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:33275 (local address 127.0.0.1:46024)
0926 23:16:32.967634 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:32.967682 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:32.967883 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:32.969339 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:32.969346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:32.969376 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:32.969888 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.969903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.972112 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.972117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.974189 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.974200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.975467 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.975473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.975816 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.975875 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 23:16:32.976015 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 23:16:32.976212 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":104,"thread_start_us":1001,"threads_started":1}
I20260926 23:16:32.979081 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.967594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:36969 (local address 127.0.0.1:43508)
0926 23:16:32.970248 (+ 2654us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:32.970264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:32.970372 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:32.971027 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:32.971032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:32.971060 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:32.971579 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.971591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.973406 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.973413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.975783 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.975798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.978272 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.978279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.978448 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.978484 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:16:32.978588 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 23:16:32.978728 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2540,"mutex_wait_us":70,"thread_start_us":1958,"threads_started":1}
I20260926 23:16:32.979403 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.967703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.16.62.66:36023)
0926 23:16:32.968927 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:32.968934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:32.969999 (+ 1065us) server_negotiation.cc:407] Connection header received
0926 23:16:32.970294 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:32.970301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:32.970518 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:32.970695 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:32.971559 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.973014 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.975936 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.978087 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.978222 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.978402 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:32.978954 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 23:16:32.979135 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":985,"threads_started":1}
I20260926 23:16:32.979146 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.968202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.16.62.65:36969)
0926 23:16:32.969383 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:32.969390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:32.970482 (+ 1092us) server_negotiation.cc:407] Connection header received
0926 23:16:32.970579 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:32.970589 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:32.970823 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:32.971005 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:32.971774 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.973206 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.976246 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.978067 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.978209 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.978405 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:32.978625 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 23:16:32.978787 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":926,"threads_started":1}
I20260926 23:16:32.980104 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:32.967150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36023 (local address 127.0.0.1:51876)
0926 23:16:32.969888 (+ 2738us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:32.969905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:32.970110 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:32.970905 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:32.970913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:32.970942 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:32.971403 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.971413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.973166 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.973171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.975792 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:32.975800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:32.978546 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:32.978552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:32.978743 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:32.978773 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:16:32.979630 (+ 857us) client_negotiation.cc:240] Negotiation successful
0926 23:16:32.979747 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2549,"mutex_wait_us":148,"thread_start_us":2840,"threads_started":1}
I20260926 23:16:32.981001 18804 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2765d0c02b4c7dac9e7d3ed84cccea (DEFAULT_TABLE table=test-table [id=bfffdb61363743c6ae5230cbbdd61560]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:32.982702 18804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2765d0c02b4c7dac9e7d3ed84cccea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:32.982445 18526 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2765d0c02b4c7dac9e7d3ed84cccea (DEFAULT_TABLE table=test-table [id=bfffdb61363743c6ae5230cbbdd61560]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:32.983508 18526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2765d0c02b4c7dac9e7d3ed84cccea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:32.983453 18665 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2765d0c02b4c7dac9e7d3ed84cccea (DEFAULT_TABLE table=test-table [id=bfffdb61363743c6ae5230cbbdd61560]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:32.984684 18665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2765d0c02b4c7dac9e7d3ed84cccea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:32.994006 18893 tablet_bootstrap.cc:492] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: Bootstrap starting.
I20260926 23:16:32.997056 18894 tablet_bootstrap.cc:492] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: Bootstrap starting.
I20260926 23:16:32.998101 18893 tablet_bootstrap.cc:654] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:32.999606 18893 log.cc:824] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:33.000623 18894 tablet_bootstrap.cc:654] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:33.002048 18894 log.cc:824] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:33.004216 18895 tablet_bootstrap.cc:492] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: Bootstrap starting.
I20260926 23:16:33.005817 18894 tablet_bootstrap.cc:492] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: No bootstrap required, opened a new log
I20260926 23:16:33.006131 18894 ts_tablet_manager.cc:1397] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 23:16:33.006668 18895 tablet_bootstrap.cc:654] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:33.007715 18895 log.cc:824] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:33.009379 18893 tablet_bootstrap.cc:492] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: No bootstrap required, opened a new log
I20260926 23:16:33.009649 18893 ts_tablet_manager.cc:1397] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260926 23:16:33.011598 18895 tablet_bootstrap.cc:492] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: No bootstrap required, opened a new log
I20260926 23:16:33.011831 18895 ts_tablet_manager.cc:1397] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 23:16:33.014374 18894 raft_consensus.cc:348] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } }
I20260926 23:16:33.014701 18894 raft_consensus.cc:374] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:33.014801 18894 raft_consensus.cc:729] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40689cb6f0814e2cae383be36c175beb, State: Initialized, Role: FOLLOWER
I20260926 23:16:33.015358 18894 consensus_queue.cc:260] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } }
I20260926 23:16:33.016386 18874 heartbeater.cc:503] Master 127.16.62.126:36325 was elected leader, sending a full tablet report...
I20260926 23:16:33.017004 18894 ts_tablet_manager.cc:1428] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 23:16:33.019980 18893 raft_consensus.cc:348] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } }
I20260926 23:16:33.020397 18893 raft_consensus.cc:374] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:33.020491 18893 raft_consensus.cc:729] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056163913b27480b830b03784577d190, State: Initialized, Role: FOLLOWER
I20260926 23:16:33.021070 18893 consensus_queue.cc:260] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } }
I20260926 23:16:33.022444 18895 raft_consensus.cc:348] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } }
I20260926 23:16:33.022722 18893 ts_tablet_manager.cc:1428] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260926 23:16:33.022837 18895 raft_consensus.cc:374] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:33.022935 18895 raft_consensus.cc:729] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5f3ec0282945cfbe01b1f3cc059e7f, State: Initialized, Role: FOLLOWER
I20260926 23:16:33.023476 18895 consensus_queue.cc:260] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } }
I20260926 23:16:33.024399 18735 heartbeater.cc:503] Master 127.16.62.126:36325 was elected leader, sending a full tablet report...
I20260926 23:16:33.024880 18895 ts_tablet_manager.cc:1428] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260926 23:16:33.117745 18736 tablet.cc:2369] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:16:33.127990 18597 tablet.cc:2369] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:16:33.143745 18875 tablet.cc:2369] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:16:33.260183 18900 raft_consensus.cc:482] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:33.260514 18900 raft_consensus.cc:504] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } }
I20260926 23:16:33.262497 18900 leader_election.cc:288] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275), cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66:36023)
I20260926 23:16:33.270853 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.263152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:53799 (local address 127.16.62.67:33275)
0926 23:16:33.263385 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.263394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.264169 (+ 775us) server_negotiation.cc:407] Connection header received
0926 23:16:33.264490 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.264495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.264573 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.264620 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:33.265580 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.266726 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.269125 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.269877 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.270061 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.270134 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.270448 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.270583 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:16:33.271210 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.262930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36023 (local address 127.16.62.65:45303)
0926 23:16:33.264292 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:33.264308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:33.264417 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:33.264778 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:33.264783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:33.264809 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:33.265432 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.265439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.266457 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.266465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.268776 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.268790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.270448 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.270454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.270616 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.270648 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:16:33.270736 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 23:16:33.270852 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"mutex_wait_us":146,"thread_start_us":661,"threads_started":1}
I20260926 23:16:33.271222 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.263123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:45303 (local address 127.16.62.66:36023)
0926 23:16:33.263376 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.263387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.264404 (+ 1017us) server_negotiation.cc:407] Connection header received
0926 23:16:33.264545 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.264551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.264635 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.264749 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:33.265599 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.266330 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.268946 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.270238 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.270409 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.270493 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.270784 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.270927 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:16:33.271732 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.262739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:33275 (local address 127.16.62.65:53799)
0926 23:16:33.263978 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:33.264000 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:33.264319 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:33.264803 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:33.264806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:33.264821 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:33.265408 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.265426 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.266881 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.266886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.268997 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.269006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.270073 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.270079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.270248 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.270270 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:16:33.271226 (+ 956us) client_negotiation.cc:240] Negotiation successful
0926 23:16:33.271384 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":45,"thread_start_us":1179,"threads_started":1}
I20260926 23:16:33.272535 18685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2765d0c02b4c7dac9e7d3ed84cccea" candidate_uuid: "056163913b27480b830b03784577d190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" is_pre_election: true
I20260926 23:16:33.273041 18685 raft_consensus.cc:2454] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056163913b27480b830b03784577d190 in term 0.
I20260926 23:16:33.272936 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2765d0c02b4c7dac9e7d3ed84cccea" candidate_uuid: "056163913b27480b830b03784577d190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb" is_pre_election: true
I20260926 23:16:33.273470 18828 raft_consensus.cc:2454] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056163913b27480b830b03784577d190 in term 0.
I20260926 23:16:33.273690 18479 leader_election.cc:302] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056163913b27480b830b03784577d190, cd5f3ec0282945cfbe01b1f3cc059e7f; no voters:
I20260926 23:16:33.274156 18900 raft_consensus.cc:2781] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:33.274325 18900 raft_consensus.cc:482] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:33.274443 18900 raft_consensus.cc:3037] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:33.276777 18900 raft_consensus.cc:504] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } }
I20260926 23:16:33.277426 18900 leader_election.cc:288] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [CANDIDATE]: Term 1 election: Requested vote from peers 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275), cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66:36023)
I20260926 23:16:33.277838 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2765d0c02b4c7dac9e7d3ed84cccea" candidate_uuid: "056163913b27480b830b03784577d190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb"
I20260926 23:16:33.278031 18828 raft_consensus.cc:3037] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:33.277971 18685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2765d0c02b4c7dac9e7d3ed84cccea" candidate_uuid: "056163913b27480b830b03784577d190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f"
I20260926 23:16:33.278203 18685 raft_consensus.cc:3037] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:33.280220 18828 raft_consensus.cc:2454] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056163913b27480b830b03784577d190 in term 1.
I20260926 23:16:33.280409 18685 raft_consensus.cc:2454] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056163913b27480b830b03784577d190 in term 1.
I20260926 23:16:33.280722 18476 leader_election.cc:302] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056163913b27480b830b03784577d190, 40689cb6f0814e2cae383be36c175beb; no voters:
I20260926 23:16:33.281075 18900 raft_consensus.cc:2781] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:33.281450 18900 raft_consensus.cc:686] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 1 LEADER]: Becoming Leader. State: Replica: 056163913b27480b830b03784577d190, State: Running, Role: LEADER
I20260926 23:16:33.281888 18900 consensus_queue.cc:237] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } }
I20260926 23:16:33.286209 18388 catalog_manager.cc:5495] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 056163913b27480b830b03784577d190 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "056163913b27480b830b03784577d190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } health_report { overall_health: HEALTHY } } }
Sep 26 23:16:33 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464591, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:33.377327 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.362690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36325 (local address 127.0.0.1:46126)
0926 23:16:33.363223 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:33.363237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:33.363362 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:33.363705 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:33.363709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:33.364469 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:33.364720 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.364729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.366123 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.366127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.367074 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.367084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.368080 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.368084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.368270 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.368990 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:33.369008 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:33.371272 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:33.374767 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:33.374776 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:33.374782 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:33.375170 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:33.375544 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:33.375549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:33.375552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:33.375672 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:33.376323 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:33.376334 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:33.376521 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 23:16:33.376654 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 23:16:33.376918 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":287,"threads_started":1}
I20260926 23:16:33.377417 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.362642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.16.62.126:36325)
0926 23:16:33.362832 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.362836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.363324 (+ 488us) server_negotiation.cc:407] Connection header received
0926 23:16:33.363458 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.363462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.363544 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.363626 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:33.364881 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.365970 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.367210 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.367885 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.368002 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.371451 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:33.371460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:33.371464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:33.371502 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:33.374637 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:33.375281 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:33.375287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:33.375291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:33.375429 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:33.375788 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:33.375793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:33.375796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:33.376193 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:33.376325 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.376767 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.377074 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":76}
I20260926 23:16:33.382380 18388 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:46126:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 23:16:33.383404 18388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:33.389760 18665 tablet_service.cc:1467] Processing CreateTablet for tablet 31569f8c872445c696dd096bdf9b3df6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32c8caadb5e5405bba8ecfd06b945f75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:16:33.389843 18804 tablet_service.cc:1467] Processing CreateTablet for tablet 31569f8c872445c696dd096bdf9b3df6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32c8caadb5e5405bba8ecfd06b945f75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:16:33.389616 18526 tablet_service.cc:1467] Processing CreateTablet for tablet 31569f8c872445c696dd096bdf9b3df6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32c8caadb5e5405bba8ecfd06b945f75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:16:33.390516 18665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31569f8c872445c696dd096bdf9b3df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:33.390534 18526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31569f8c872445c696dd096bdf9b3df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:33.390530 18804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31569f8c872445c696dd096bdf9b3df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:33.396214 18895 tablet_bootstrap.cc:492] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: Bootstrap starting.
I20260926 23:16:33.397367 18893 tablet_bootstrap.cc:492] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: Bootstrap starting.
I20260926 23:16:33.397598 18895 tablet_bootstrap.cc:654] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:33.398192 18894 tablet_bootstrap.cc:492] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: Bootstrap starting.
I20260926 23:16:33.399179 18893 tablet_bootstrap.cc:654] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:33.399713 18894 tablet_bootstrap.cc:654] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:33.401624 18893 tablet_bootstrap.cc:492] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: No bootstrap required, opened a new log
I20260926 23:16:33.401767 18893 ts_tablet_manager.cc:1397] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260926 23:16:33.402026 18894 tablet_bootstrap.cc:492] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: No bootstrap required, opened a new log
I20260926 23:16:33.402154 18894 ts_tablet_manager.cc:1397] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 23:16:33.402470 18893 raft_consensus.cc:348] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.402778 18895 tablet_bootstrap.cc:492] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: No bootstrap required, opened a new log
I20260926 23:16:33.402756 18893 raft_consensus.cc:374] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:33.402866 18893 raft_consensus.cc:729] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056163913b27480b830b03784577d190, State: Initialized, Role: FOLLOWER
I20260926 23:16:33.402793 18894 raft_consensus.cc:348] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.403026 18895 ts_tablet_manager.cc:1397] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260926 23:16:33.403128 18894 raft_consensus.cc:374] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:33.403137 18893 consensus_queue.cc:260] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.403286 18894 raft_consensus.cc:729] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40689cb6f0814e2cae383be36c175beb, State: Initialized, Role: FOLLOWER
I20260926 23:16:33.403714 18893 ts_tablet_manager.cc:1428] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 23:16:33.403527 18894 consensus_queue.cc:260] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.403842 18895 raft_consensus.cc:348] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.404199 18895 raft_consensus.cc:374] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:33.404065 18900 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.404350 18900 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:33.404373 18895 raft_consensus.cc:729] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5f3ec0282945cfbe01b1f3cc059e7f, State: Initialized, Role: FOLLOWER
I20260926 23:16:33.404659 18895 consensus_queue.cc:260] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.404784 18899 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.405114 18899 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:33.405232 18901 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.405587 18894 ts_tablet_manager.cc:1428] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 23:16:33.405586 18901 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:33.406641 18895 ts_tablet_manager.cc:1428] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 23:16:33.481019 18901 raft_consensus.cc:482] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:33.481215 18901 raft_consensus.cc:504] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.482646 18901 leader_election.cc:288] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 056163913b27480b830b03784577d190 (127.16.62.65:36969), 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275)
I20260926 23:16:33.489764 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.483042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:37681 (local address 127.16.62.67:33275)
0926 23:16:33.483229 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.483235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.484475 (+ 1240us) server_negotiation.cc:407] Connection header received
0926 23:16:33.484635 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.484639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.484700 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.484768 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:33.485290 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.486311 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.488001 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.488825 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.489004 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.489082 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.489369 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.489484 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:16:33.489878 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.483098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:60997 (local address 127.16.62.65:36969)
0926 23:16:33.484049 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.484057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.484075 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:16:33.484185 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.484189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.484262 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.484342 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:33.484993 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.486058 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.488012 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.488838 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.489013 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.489096 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.489402 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.489529 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":680,"threads_started":1}
I20260926 23:16:33.489923 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.483020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:33275 (local address 127.16.62.66:37681)
0926 23:16:33.484368 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:33.484384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:33.484500 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:33.484816 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:33.484825 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:33.484846 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:33.485147 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.485154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.486435 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.486440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.487857 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.487870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.489056 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.489062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.489216 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.489240 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 23:16:33.489336 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:16:33.489457 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"mutex_wait_us":54,"thread_start_us":323,"threads_started":1}
I20260926 23:16:33.489975 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.483020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:36969 (local address 127.16.62.66:60997)
0926 23:16:33.483791 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:33.483814 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:33.483960 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:33.484393 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:33.484402 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:33.484433 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:33.484837 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.484847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.486189 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.486218 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.487855 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.487866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.489109 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.489118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.489254 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.489276 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:16:33.489378 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 23:16:33.489498 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":27,"thread_start_us":923,"threads_started":1}
I20260926 23:16:33.490679 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31569f8c872445c696dd096bdf9b3df6" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb" is_pre_election: true
I20260926 23:16:33.490962 18828 raft_consensus.cc:2454] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 0.
I20260926 23:16:33.490962 18548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31569f8c872445c696dd096bdf9b3df6" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056163913b27480b830b03784577d190" is_pre_election: true
I20260926 23:16:33.491375 18548 raft_consensus.cc:2454] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 0.
I20260926 23:16:33.491604 18616 leader_election.cc:302] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40689cb6f0814e2cae383be36c175beb, cd5f3ec0282945cfbe01b1f3cc059e7f; no voters:
I20260926 23:16:33.492029 18901 raft_consensus.cc:2781] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:33.492193 18901 raft_consensus.cc:482] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:33.492298 18901 raft_consensus.cc:3037] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:33.494517 18901 raft_consensus.cc:504] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.495038 18901 leader_election.cc:288] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 election: Requested vote from peers 056163913b27480b830b03784577d190 (127.16.62.65:36969), 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275)
I20260926 23:16:33.495456 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31569f8c872445c696dd096bdf9b3df6" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb"
I20260926 23:16:33.495443 18548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31569f8c872445c696dd096bdf9b3df6" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056163913b27480b830b03784577d190"
I20260926 23:16:33.495644 18548 raft_consensus.cc:3037] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:33.495661 18828 raft_consensus.cc:3037] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:33.498757 18548 raft_consensus.cc:2454] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 1.
I20260926 23:16:33.498734 18828 raft_consensus.cc:2454] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 1.
I20260926 23:16:33.499190 18618 leader_election.cc:302] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056163913b27480b830b03784577d190, cd5f3ec0282945cfbe01b1f3cc059e7f; no voters:
I20260926 23:16:33.499526 18901 raft_consensus.cc:2781] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:33.500002 18901 raft_consensus.cc:686] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 LEADER]: Becoming Leader. State: Replica: cd5f3ec0282945cfbe01b1f3cc059e7f, State: Running, Role: LEADER
I20260926 23:16:33.500556 18901 consensus_queue.cc:237] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.503876 18918 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: TxnStatusTablet state changed. Reason: New leader cd5f3ec0282945cfbe01b1f3cc059e7f. Latest consensus state: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.504328 18918 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:16:33.505004 18920 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 23:16:33.505265 18920 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 23:16:33.505700 18387 catalog_manager.cc:5495] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f reported cstate change: term changed from 0 to 1, leader changed from <none> to cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66). New cstate: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } health_report { overall_health: UNKNOWN } } }
I20260926 23:16:33.529572 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.523547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36023 (local address 127.0.0.1:51884)
0926 23:16:33.523783 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:33.523803 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:33.523913 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:33.524425 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:33.524429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:33.524438 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:16:33.524591 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.524598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.526003 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.526007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.526931 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.526941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.527890 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.527894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.528043 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.528089 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.529102 (+ 1013us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.529118 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 23:16:33.529190 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 23:16:33.529294 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 23:16:33.529623 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.523647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.16.62.66:36023)
0926 23:16:33.523829 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.523837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.523861 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:16:33.524095 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.524101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.524265 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.524411 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:33.524741 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.525870 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.527061 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.527746 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.527876 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.528327 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.528995 (+ 668us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.529136 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.529213 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.529313 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:16:33.536757 18828 raft_consensus.cc:1261] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Refusing update from remote peer cd5f3ec0282945cfbe01b1f3cc059e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:33.536787 18548 raft_consensus.cc:1261] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Refusing update from remote peer cd5f3ec0282945cfbe01b1f3cc059e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:33.537657 18918 consensus_queue.cc:1035] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:33.538132 18901 consensus_queue.cc:1035] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:33.550282 18899 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: TxnStatusTablet state changed. Reason: New leader cd5f3ec0282945cfbe01b1f3cc059e7f. Latest consensus state: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.550557 18899 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:33.551745 18900 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: TxnStatusTablet state changed. Reason: New leader cd5f3ec0282945cfbe01b1f3cc059e7f. Latest consensus state: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.552050 18900 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:33.554068 18923 mvcc.cc:205] Tried to move back new op lower bound from 7333742975117185024 to 7333742974981713920. Current Snapshot: MvccSnapshot[applied={T|T < 7333742975117185024}]
I20260926 23:16:33.563906 18918 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.564272 18918 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:16:33.568457 18901 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.568730 18901 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P cd5f3ec0282945cfbe01b1f3cc059e7f: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:16:33.575968 18900 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.576260 18900 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P 056163913b27480b830b03784577d190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:33.581558 18899 tablet_replica.cc:439] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } } }
I20260926 23:16:33.581835 18899 tablet_replica.cc:442] T 31569f8c872445c696dd096bdf9b3df6 P 40689cb6f0814e2cae383be36c175beb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:16:33.589258 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.581503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.16.62.65:36969)
0926 23:16:33.581711 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.581720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.584110 (+ 2390us) server_negotiation.cc:407] Connection header received
0926 23:16:33.584391 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.584395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.584470 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.584615 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:33.585048 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.585789 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.587229 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.587621 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.587760 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.588208 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.588688 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.588837 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.588932 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.589008 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 23:16:33.605989 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.599901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36023 (local address 127.16.62.65:45367)
0926 23:16:33.600421 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:33.600436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:33.600543 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:33.600877 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:33.600881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:33.600889 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:16:33.601165 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.601172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.602701 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.602704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.603957 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.603968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.604927 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.604932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.605069 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.605117 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.605529 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.605536 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 23:16:33.605598 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 23:16:33.605693 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":387,"threads_started":1}
I20260926 23:16:33.606035 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.600065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:45367 (local address 127.16.62.66:36023)
0926 23:16:33.600272 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.600280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.600516 (+ 236us) server_negotiation.cc:407] Connection header received
0926 23:16:33.600671 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.600677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.600765 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.600860 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:33.601289 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.602535 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.604081 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.604775 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.604895 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.605305 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.605433 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.605520 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.605657 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.605748 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 23:16:33.615057 18828 raft_consensus.cc:1261] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Refusing update from remote peer 056163913b27480b830b03784577d190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:33.615093 18685 raft_consensus.cc:1261] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Refusing update from remote peer 056163913b27480b830b03784577d190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:33.615742 18900 consensus_queue.cc:1035] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:33.616307 18904 consensus_queue.cc:1035] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:33.626052 18925 mvcc.cc:205] Tried to move back new op lower bound from 7333742975440801792 to 7333742974086402048. Current Snapshot: MvccSnapshot[applied={T|T < 7333742975440801792}]
I20260926 23:16:33.657320 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.650842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:46877 (local address 127.16.62.65:36969)
0926 23:16:33.651032 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.651038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.651840 (+ 802us) server_negotiation.cc:407] Connection header received
0926 23:16:33.652019 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.652024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.652106 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.652281 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:33.652824 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.653560 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.655282 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.655949 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.656082 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.656696 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.656802 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.656872 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.657024 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.657095 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:16:33.657580 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.650867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:36969 (local address 127.16.62.66:46877)
0926 23:16:33.651710 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:33.651748 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:33.651891 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:33.652296 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:33.652300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:33.652312 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:16:33.652681 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.652695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.653680 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.653688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.655123 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.655137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.656264 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.656271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.656447 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.656517 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.656901 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.656912 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:16:33.656996 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 23:16:33.657154 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":469,"threads_started":1}
I20260926 23:16:33.698002 18387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:33.698292 18387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:33.702231 18387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc: Sending DeleteTablet for 3 replicas of tablet 5f2765d0c02b4c7dac9e7d3ed84cccea
I20260926 23:16:33.703320 18665 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2765d0c02b4c7dac9e7d3ed84cccea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:33 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:51876
I20260926 23:16:33.703329 18804 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2765d0c02b4c7dac9e7d3ed84cccea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:33 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:46024
I20260926 23:16:33.705746 18526 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2765d0c02b4c7dac9e7d3ed84cccea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:33 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:43508
I20260926 23:16:33.712759 18945 tablet_replica.cc:330] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: stopping tablet replica
I20260926 23:16:33.713317 18945 raft_consensus.cc:2229] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:33.713740 18945 raft_consensus.cc:2258] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:33.714218 18388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:16:33.715035 18388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:33.715884 18945 ts_tablet_manager.cc:1905] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:33.722931 18804 tablet_service.cc:1467] Processing CreateTablet for tablet 8294c0400a8a4484b4fb7f923a406372 (DEFAULT_TABLE table=test-table [id=81a18d332b9645cab51929f239ebb063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:33.723349 18804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8294c0400a8a4484b4fb7f923a406372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:33.729152 18946 tablet_replica.cc:330] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: stopping tablet replica
I20260926 23:16:33.729382 18664 tablet_service.cc:1467] Processing CreateTablet for tablet 8294c0400a8a4484b4fb7f923a406372 (DEFAULT_TABLE table=test-table [id=81a18d332b9645cab51929f239ebb063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:33.729835 18946 raft_consensus.cc:2229] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:16:33.730002 18664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8294c0400a8a4484b4fb7f923a406372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:33.730692 18946 raft_consensus.cc:2258] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:33.732693 18946 ts_tablet_manager.cc:1905] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:33.733302 18944 tablet_replica.cc:330] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: stopping tablet replica
I20260926 23:16:33.733762 18944 raft_consensus.cc:2229] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:33.734210 18944 raft_consensus.cc:2258] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:33.737465 18526 tablet_service.cc:1467] Processing CreateTablet for tablet 8294c0400a8a4484b4fb7f923a406372 (DEFAULT_TABLE table=test-table [id=81a18d332b9645cab51929f239ebb063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:33.737991 18526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8294c0400a8a4484b4fb7f923a406372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:33.740890 18895 tablet_bootstrap.cc:492] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: Bootstrap starting.
I20260926 23:16:33.743041 18894 tablet_bootstrap.cc:492] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: Bootstrap starting.
I20260926 23:16:33.743574 18895 tablet_bootstrap.cc:654] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:33.744762 18894 tablet_bootstrap.cc:654] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:33.750489 18944 ts_tablet_manager.cc:1905] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:33.750875 18893 tablet_bootstrap.cc:492] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: Bootstrap starting.
I20260926 23:16:33.752516 18893 tablet_bootstrap.cc:654] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:33.763285 18945 ts_tablet_manager.cc:1918] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:16:33.763514 18945 log.cc:1196] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/5f2765d0c02b4c7dac9e7d3ed84cccea
I20260926 23:16:33.763516 18946 ts_tablet_manager.cc:1918] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:16:33.763656 18946 log.cc:1196] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/5f2765d0c02b4c7dac9e7d3ed84cccea
I20260926 23:16:33.763581 18894 tablet_bootstrap.cc:492] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: No bootstrap required, opened a new log
I20260926 23:16:33.762482 18895 tablet_bootstrap.cc:492] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: No bootstrap required, opened a new log
I20260926 23:16:33.763821 18894 ts_tablet_manager.cc:1397] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20260926 23:16:33.764068 18895 ts_tablet_manager.cc:1397] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.003s
I20260926 23:16:33.764626 18894 raft_consensus.cc:348] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.764874 18894 raft_consensus.cc:374] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:33.764961 18894 raft_consensus.cc:729] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40689cb6f0814e2cae383be36c175beb, State: Initialized, Role: FOLLOWER
I20260926 23:16:33.765126 18895 raft_consensus.cc:348] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.765280 18894 consensus_queue.cc:260] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.765545 18895 raft_consensus.cc:374] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:33.765673 18895 raft_consensus.cc:729] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5f3ec0282945cfbe01b1f3cc059e7f, State: Initialized, Role: FOLLOWER
I20260926 23:16:33.765980 18895 consensus_queue.cc:260] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.768599 18894 ts_tablet_manager.cc:1428] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 23:16:33.769145 18895 ts_tablet_manager.cc:1428] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260926 23:16:33.771474 18945 ts_tablet_manager.cc:1939] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 40689cb6f0814e2cae383be36c175beb: Deleting consensus metadata
I20260926 23:16:33.773211 18373 catalog_manager.cc:4841] TS 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275): tablet 5f2765d0c02b4c7dac9e7d3ed84cccea (table test-table [id=bfffdb61363743c6ae5230cbbdd61560]) successfully deleted
I20260926 23:16:33.764954 18893 tablet_bootstrap.cc:492] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: No bootstrap required, opened a new log
I20260926 23:16:33.777441 18893 ts_tablet_manager.cc:1397] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.004s
I20260926 23:16:33.778148 18893 raft_consensus.cc:348] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.778371 18893 raft_consensus.cc:374] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:33.778448 18893 raft_consensus.cc:729] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056163913b27480b830b03784577d190, State: Initialized, Role: FOLLOWER
I20260926 23:16:33.778656 18893 consensus_queue.cc:260] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.779171 18893 ts_tablet_manager.cc:1428] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 23:16:33.781224 18944 ts_tablet_manager.cc:1918] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:16:33.781419 18944 log.cc:1196] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/5f2765d0c02b4c7dac9e7d3ed84cccea
I20260926 23:16:33.781972 18944 ts_tablet_manager.cc:1939] T 5f2765d0c02b4c7dac9e7d3ed84cccea P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting consensus metadata
I20260926 23:16:33.783475 18946 ts_tablet_manager.cc:1939] T 5f2765d0c02b4c7dac9e7d3ed84cccea P 056163913b27480b830b03784577d190: Deleting consensus metadata
I20260926 23:16:33.783572 18376 catalog_manager.cc:4841] TS cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66:36023): tablet 5f2765d0c02b4c7dac9e7d3ed84cccea (table test-table [id=bfffdb61363743c6ae5230cbbdd61560]) successfully deleted
I20260926 23:16:33.785118 18375 catalog_manager.cc:4841] TS 056163913b27480b830b03784577d190 (127.16.62.65:36969): tablet 5f2765d0c02b4c7dac9e7d3ed84cccea (table test-table [id=bfffdb61363743c6ae5230cbbdd61560]) successfully deleted
I20260926 23:16:33.876533 18901 raft_consensus.cc:482] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:33.876767 18901 raft_consensus.cc:504] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.877552 18901 leader_election.cc:288] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 056163913b27480b830b03784577d190 (127.16.62.65:36969), 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275)
I20260926 23:16:33.878654 18548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8294c0400a8a4484b4fb7f923a406372" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056163913b27480b830b03784577d190" is_pre_election: true
I20260926 23:16:33.878917 18548 raft_consensus.cc:2454] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 0.
I20260926 23:16:33.880230 18618 leader_election.cc:302] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056163913b27480b830b03784577d190, cd5f3ec0282945cfbe01b1f3cc059e7f; no voters:
I20260926 23:16:33.880301 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8294c0400a8a4484b4fb7f923a406372" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb" is_pre_election: true
I20260926 23:16:33.880537 18828 raft_consensus.cc:2454] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 0.
I20260926 23:16:33.880672 18901 raft_consensus.cc:2781] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:33.880793 18901 raft_consensus.cc:482] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:33.880899 18901 raft_consensus.cc:3037] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:33.884047 18901 raft_consensus.cc:504] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.885186 18548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8294c0400a8a4484b4fb7f923a406372" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056163913b27480b830b03784577d190"
I20260926 23:16:33.885406 18548 raft_consensus.cc:3037] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:33.888705 18548 raft_consensus.cc:2454] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 1.
I20260926 23:16:33.889263 18618 leader_election.cc:302] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056163913b27480b830b03784577d190, cd5f3ec0282945cfbe01b1f3cc059e7f; no voters:
I20260926 23:16:33.889597 18928 raft_consensus.cc:2781] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:33.889740 18928 raft_consensus.cc:686] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 LEADER]: Becoming Leader. State: Replica: cd5f3ec0282945cfbe01b1f3cc059e7f, State: Running, Role: LEADER
I20260926 23:16:33.890125 18928 consensus_queue.cc:237] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:33.899140 18901 leader_election.cc:288] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 election: Requested vote from peers 056163913b27480b830b03784577d190 (127.16.62.65:36969), 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275)
I20260926 23:16:33.899755 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8294c0400a8a4484b4fb7f923a406372" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb"
I20260926 23:16:33.900002 18828 raft_consensus.cc:3037] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:33.903338 18828 raft_consensus.cc:2454] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 1.
I20260926 23:16:33.918135 18388 catalog_manager.cc:5495] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f reported cstate change: term changed from 0 to 1, leader changed from <none> to cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66). New cstate: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } health_report { overall_health: UNKNOWN } } }
I20260926 23:16:33.955231 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.948911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.16.62.66:36023)
0926 23:16:33.949089 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.949094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.949112 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:16:33.949283 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.949289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.949375 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.949477 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:33.949882 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.951131 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.953159 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.953847 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.953959 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.954488 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.954645 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.954733 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.954875 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.954947 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 23:16:33.970037 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.963907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:55695 (local address 127.16.62.66:36023)
0926 23:16:33.964058 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:33.964064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:33.964161 (+ 97us) server_negotiation.cc:407] Connection header received
0926 23:16:33.964327 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:33.964332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:33.964408 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:33.964496 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:33.964859 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.965916 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.967863 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.968567 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.968722 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.969199 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.969340 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.969437 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:33.969598 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 23:16:33.969679 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 23:16:33.970135 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:33.963825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36023 (local address 127.16.62.66:55695)
0926 23:16:33.964060 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:33.964077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:33.964212 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:33.964522 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:33.964527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:33.964540 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:16:33.964724 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.964733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.966046 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.966051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.967687 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:33.967701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:33.968769 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:33.968777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:33.968966 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:33.969015 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:16:33.969455 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:16:33.969465 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:16:33.969565 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 23:16:33.969708 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":1}
I20260926 23:16:33.980798 18548 raft_consensus.cc:1261] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Refusing update from remote peer cd5f3ec0282945cfbe01b1f3cc059e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:33.980844 18828 raft_consensus.cc:1261] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Refusing update from remote peer cd5f3ec0282945cfbe01b1f3cc059e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:33.981331 18901 consensus_queue.cc:1035] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:33.981745 18901 consensus_queue.cc:1035] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:34.000010 18924 mvcc.cc:205] Tried to move back new op lower bound from 7333742976940756992 to 7333742976577347584. Current Snapshot: MvccSnapshot[applied={T|T < 7333742976940756992}]
W20260926 23:16:34.022028 18923 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 23:16:35.610839 18388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:35.611027 18388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:35.614589 18388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc: Sending DeleteTablet for 3 replicas of tablet 8294c0400a8a4484b4fb7f923a406372
I20260926 23:16:35.615249 18526 tablet_service.cc:1514] Processing DeleteTablet for tablet 8294c0400a8a4484b4fb7f923a406372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:35 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:43508
I20260926 23:16:35.615339 18664 tablet_service.cc:1514] Processing DeleteTablet for tablet 8294c0400a8a4484b4fb7f923a406372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:35 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:51876
I20260926 23:16:35.615429 18804 tablet_service.cc:1514] Processing DeleteTablet for tablet 8294c0400a8a4484b4fb7f923a406372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:35 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:46024
I20260926 23:16:35.626271 18975 tablet_replica.cc:330] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: stopping tablet replica
I20260926 23:16:35.626663 18975 raft_consensus.cc:2229] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:16:35.627313 18976 tablet_replica.cc:330] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: stopping tablet replica
I20260926 23:16:35.627372 18975 raft_consensus.cc:2258] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:35.627481 18974 tablet_replica.cc:330] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: stopping tablet replica
I20260926 23:16:35.627585 18976 raft_consensus.cc:2229] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:35.627692 18974 raft_consensus.cc:2229] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:35.627887 18976 raft_consensus.cc:2258] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:35.627949 18974 raft_consensus.cc:2258] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:35.628449 18975 ts_tablet_manager.cc:1905] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:35.628875 18976 ts_tablet_manager.cc:1905] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:35.628880 18974 ts_tablet_manager.cc:1905] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:35.633519 18975 ts_tablet_manager.cc:1918] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:16:35.633734 18975 log.cc:1196] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/8294c0400a8a4484b4fb7f923a406372
I20260926 23:16:35.634207 18975 ts_tablet_manager.cc:1939] T 8294c0400a8a4484b4fb7f923a406372 P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting consensus metadata
I20260926 23:16:35.635145 18976 ts_tablet_manager.cc:1918] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:16:35.635309 18976 log.cc:1196] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/8294c0400a8a4484b4fb7f923a406372
I20260926 23:16:35.635356 18974 ts_tablet_manager.cc:1918] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:16:35.635321 18376 catalog_manager.cc:4841] TS cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66:36023): tablet 8294c0400a8a4484b4fb7f923a406372 (table test-table [id=81a18d332b9645cab51929f239ebb063]) successfully deleted
I20260926 23:16:35.635490 18974 log.cc:1196] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/8294c0400a8a4484b4fb7f923a406372
I20260926 23:16:35.635831 18976 ts_tablet_manager.cc:1939] T 8294c0400a8a4484b4fb7f923a406372 P 40689cb6f0814e2cae383be36c175beb: Deleting consensus metadata
I20260926 23:16:35.635963 18974 ts_tablet_manager.cc:1939] T 8294c0400a8a4484b4fb7f923a406372 P 056163913b27480b830b03784577d190: Deleting consensus metadata
I20260926 23:16:35.636898 18375 catalog_manager.cc:4841] TS 056163913b27480b830b03784577d190 (127.16.62.65:36969): tablet 8294c0400a8a4484b4fb7f923a406372 (table test-table [id=81a18d332b9645cab51929f239ebb063]) successfully deleted
I20260926 23:16:35.636942 18373 catalog_manager.cc:4841] TS 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275): tablet 8294c0400a8a4484b4fb7f923a406372 (table test-table [id=81a18d332b9645cab51929f239ebb063]) successfully deleted
Sep 26 23:16:35 dist-test-slave-vcv1 krb5kdc[18344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:16:35 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:35.674763 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:35.659690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.16.62.126:36325)
0926 23:16:35.660285 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:35.660289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:35.660349 (+ 60us) server_negotiation.cc:407] Connection header received
0926 23:16:35.660505 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:35.660509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:35.660573 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:35.660671 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:35.661935 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:35.662736 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:35.664488 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:35.665063 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:35.665189 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:35.669717 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:35.669725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:35.669730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:35.669768 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:35.672356 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:35.672957 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:35.672968 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:35.672971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:35.673082 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:35.673431 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:35.673436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:35.673441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:35.673706 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:35.673940 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:35.674216 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 23:16:35.674419 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":289,"threads_started":1}
I20260926 23:16:35.678323 18388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:16:35.678885 18388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:35.684890 18526 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce096735d984ac6aebb5ed261dd97ce (DEFAULT_TABLE table=test-table [id=34187edebba24d6facea7aa95d5a2ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:35.685043 18664 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce096735d984ac6aebb5ed261dd97ce (DEFAULT_TABLE table=test-table [id=34187edebba24d6facea7aa95d5a2ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:35.685348 18526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce096735d984ac6aebb5ed261dd97ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:35.685280 18804 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce096735d984ac6aebb5ed261dd97ce (DEFAULT_TABLE table=test-table [id=34187edebba24d6facea7aa95d5a2ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:35.685555 18664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce096735d984ac6aebb5ed261dd97ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:35.685652 18804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce096735d984ac6aebb5ed261dd97ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:35.692747 18985 tablet_bootstrap.cc:492] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: Bootstrap starting.
I20260926 23:16:35.694273 18985 tablet_bootstrap.cc:654] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:35.696623 18986 tablet_bootstrap.cc:492] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: Bootstrap starting.
I20260926 23:16:35.696702 18985 tablet_bootstrap.cc:492] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: No bootstrap required, opened a new log
I20260926 23:16:35.696820 18985 ts_tablet_manager.cc:1397] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 23:16:35.697583 18985 raft_consensus.cc:348] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.697849 18985 raft_consensus.cc:374] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:35.697960 18985 raft_consensus.cc:729] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5f3ec0282945cfbe01b1f3cc059e7f, State: Initialized, Role: FOLLOWER
I20260926 23:16:35.698267 18986 tablet_bootstrap.cc:654] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:35.696095 18987 tablet_bootstrap.cc:492] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: Bootstrap starting.
I20260926 23:16:35.698235 18985 consensus_queue.cc:260] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.698818 18985 ts_tablet_manager.cc:1428] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 23:16:35.700067 18987 tablet_bootstrap.cc:654] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:35.702159 18986 tablet_bootstrap.cc:492] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: No bootstrap required, opened a new log
I20260926 23:16:35.702349 18986 ts_tablet_manager.cc:1397] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 23:16:35.702900 18987 tablet_bootstrap.cc:492] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: No bootstrap required, opened a new log
I20260926 23:16:35.703063 18987 ts_tablet_manager.cc:1397] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 23:16:35.703243 18986 raft_consensus.cc:348] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.703461 18986 raft_consensus.cc:374] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:35.703576 18986 raft_consensus.cc:729] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056163913b27480b830b03784577d190, State: Initialized, Role: FOLLOWER
I20260926 23:16:35.703825 18987 raft_consensus.cc:348] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.703814 18986 consensus_queue.cc:260] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.704027 18987 raft_consensus.cc:374] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:35.704118 18987 raft_consensus.cc:729] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40689cb6f0814e2cae383be36c175beb, State: Initialized, Role: FOLLOWER
I20260926 23:16:35.704398 18987 consensus_queue.cc:260] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.704643 18986 ts_tablet_manager.cc:1428] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 23:16:35.705252 18987 ts_tablet_manager.cc:1428] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:16:35.729413 18991 raft_consensus.cc:482] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:35.729626 18991 raft_consensus.cc:504] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.730342 18991 leader_election.cc:288] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66:36023), 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275)
I20260926 23:16:35.730834 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce096735d984ac6aebb5ed261dd97ce" candidate_uuid: "056163913b27480b830b03784577d190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb" is_pre_election: true
I20260926 23:16:35.730867 18685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce096735d984ac6aebb5ed261dd97ce" candidate_uuid: "056163913b27480b830b03784577d190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" is_pre_election: true
I20260926 23:16:35.731062 18828 raft_consensus.cc:2454] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056163913b27480b830b03784577d190 in term 0.
I20260926 23:16:35.731101 18685 raft_consensus.cc:2454] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056163913b27480b830b03784577d190 in term 0.
I20260926 23:16:35.731540 18476 leader_election.cc:302] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056163913b27480b830b03784577d190, 40689cb6f0814e2cae383be36c175beb; no voters:
I20260926 23:16:35.731803 18991 raft_consensus.cc:2781] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:35.731921 18991 raft_consensus.cc:482] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:35.731990 18991 raft_consensus.cc:3037] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:35.734894 18991 raft_consensus.cc:504] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.735564 18991 leader_election.cc:288] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [CANDIDATE]: Term 1 election: Requested vote from peers cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66:36023), 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275)
I20260926 23:16:35.735932 18685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce096735d984ac6aebb5ed261dd97ce" candidate_uuid: "056163913b27480b830b03784577d190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f"
I20260926 23:16:35.736231 18685 raft_consensus.cc:3037] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:35.736169 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce096735d984ac6aebb5ed261dd97ce" candidate_uuid: "056163913b27480b830b03784577d190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb"
I20260926 23:16:35.736486 18828 raft_consensus.cc:3037] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:35.740316 18685 raft_consensus.cc:2454] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056163913b27480b830b03784577d190 in term 1.
I20260926 23:16:35.740710 18828 raft_consensus.cc:2454] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056163913b27480b830b03784577d190 in term 1.
I20260926 23:16:35.740931 18479 leader_election.cc:302] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056163913b27480b830b03784577d190, cd5f3ec0282945cfbe01b1f3cc059e7f; no voters:
I20260926 23:16:35.741345 18991 raft_consensus.cc:2781] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:35.744052 18991 raft_consensus.cc:686] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 1 LEADER]: Becoming Leader. State: Replica: 056163913b27480b830b03784577d190, State: Running, Role: LEADER
I20260926 23:16:35.744586 18991 consensus_queue.cc:237] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.747875 18389 catalog_manager.cc:5495] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 056163913b27480b830b03784577d190 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "056163913b27480b830b03784577d190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } health_report { overall_health: UNKNOWN } } }
I20260926 23:16:35.773634 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:35.766648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.16.62.65:36969)
0926 23:16:35.767915 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:35.767919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:35.767954 (+ 35us) server_negotiation.cc:407] Connection header received
0926 23:16:35.767999 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:35.768002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:35.768055 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:35.768130 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:35.768655 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:35.769808 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:35.771457 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:35.772075 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:35.772247 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:35.772778 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:35.772897 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:35.772964 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:35.773078 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 23:16:35.773132 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":860,"threads_started":1}
I20260926 23:16:35.784166 18685 raft_consensus.cc:1261] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Refusing update from remote peer 056163913b27480b830b03784577d190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:35.784369 18828 raft_consensus.cc:1261] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Refusing update from remote peer 056163913b27480b830b03784577d190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:35.784930 18993 consensus_queue.cc:1035] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:35.785444 18991 consensus_queue.cc:1035] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:35.839435 18389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:35.839687 18389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:35.842483 18389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc: Sending DeleteTablet for 3 replicas of tablet 7ce096735d984ac6aebb5ed261dd97ce
I20260926 23:16:35.843438 18526 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce096735d984ac6aebb5ed261dd97ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:35 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:43508
I20260926 23:16:35.843420 18664 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce096735d984ac6aebb5ed261dd97ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:35 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:51876
I20260926 23:16:35.843611 18804 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce096735d984ac6aebb5ed261dd97ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:35 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:46024
I20260926 23:16:35.843765 18974 tablet_replica.cc:330] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: stopping tablet replica
I20260926 23:16:35.843819 18975 tablet_replica.cc:330] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: stopping tablet replica
I20260926 23:16:35.843966 18976 tablet_replica.cc:330] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: stopping tablet replica
I20260926 23:16:35.844097 18974 raft_consensus.cc:2229] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:16:35.844208 18975 raft_consensus.cc:2229] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:35.844384 18976 raft_consensus.cc:2229] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:35.844547 18975 raft_consensus.cc:2258] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:35.844570 18974 raft_consensus.cc:2258] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:35.844707 18976 raft_consensus.cc:2258] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:35.844897 18389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:16:35.845830 18389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:35.845976 18974 ts_tablet_manager.cc:1905] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:35.846046 18976 ts_tablet_manager.cc:1905] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:35.846242 18975 ts_tablet_manager.cc:1905] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:35.853456 18526 tablet_service.cc:1467] Processing CreateTablet for tablet a78bbe6a7aca4ee9a15edb5d9553d43d (DEFAULT_TABLE table=test-table [id=591409bbe8a046658d0c34bda7a4709c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:35.853459 18804 tablet_service.cc:1467] Processing CreateTablet for tablet a78bbe6a7aca4ee9a15edb5d9553d43d (DEFAULT_TABLE table=test-table [id=591409bbe8a046658d0c34bda7a4709c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:35.853886 18526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78bbe6a7aca4ee9a15edb5d9553d43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:35.853880 18804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78bbe6a7aca4ee9a15edb5d9553d43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:35.854683 18975 ts_tablet_manager.cc:1918] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:16:35.855021 18975 log.cc:1196] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/7ce096735d984ac6aebb5ed261dd97ce
I20260926 23:16:35.855075 18974 ts_tablet_manager.cc:1918] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:16:35.855221 18974 log.cc:1196] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/7ce096735d984ac6aebb5ed261dd97ce
I20260926 23:16:35.855696 18975 ts_tablet_manager.cc:1939] T 7ce096735d984ac6aebb5ed261dd97ce P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting consensus metadata
I20260926 23:16:35.855716 18974 ts_tablet_manager.cc:1939] T 7ce096735d984ac6aebb5ed261dd97ce P 056163913b27480b830b03784577d190: Deleting consensus metadata
I20260926 23:16:35.856720 18375 catalog_manager.cc:4841] TS 056163913b27480b830b03784577d190 (127.16.62.65:36969): tablet 7ce096735d984ac6aebb5ed261dd97ce (table test-table [id=34187edebba24d6facea7aa95d5a2ee7]) successfully deleted
I20260926 23:16:35.857191 18376 catalog_manager.cc:4841] TS cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66:36023): tablet 7ce096735d984ac6aebb5ed261dd97ce (table test-table [id=34187edebba24d6facea7aa95d5a2ee7]) successfully deleted
I20260926 23:16:35.857290 18664 tablet_service.cc:1467] Processing CreateTablet for tablet a78bbe6a7aca4ee9a15edb5d9553d43d (DEFAULT_TABLE table=test-table [id=591409bbe8a046658d0c34bda7a4709c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:35.857683 18664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78bbe6a7aca4ee9a15edb5d9553d43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:35.860090 18986 tablet_bootstrap.cc:492] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: Bootstrap starting.
I20260926 23:16:35.860312 18987 tablet_bootstrap.cc:492] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: Bootstrap starting.
I20260926 23:16:35.861289 18976 ts_tablet_manager.cc:1918] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:16:35.861708 18976 log.cc:1196] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/7ce096735d984ac6aebb5ed261dd97ce
I20260926 23:16:35.862287 18976 ts_tablet_manager.cc:1939] T 7ce096735d984ac6aebb5ed261dd97ce P 40689cb6f0814e2cae383be36c175beb: Deleting consensus metadata
I20260926 23:16:35.862319 18986 tablet_bootstrap.cc:654] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:35.862942 18987 tablet_bootstrap.cc:654] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:35.863538 18373 catalog_manager.cc:4841] TS 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275): tablet 7ce096735d984ac6aebb5ed261dd97ce (table test-table [id=34187edebba24d6facea7aa95d5a2ee7]) successfully deleted
I20260926 23:16:35.866827 18987 tablet_bootstrap.cc:492] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: No bootstrap required, opened a new log
I20260926 23:16:35.867118 18987 ts_tablet_manager.cc:1397] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 23:16:35.867873 18985 tablet_bootstrap.cc:492] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: Bootstrap starting.
I20260926 23:16:35.868539 18987 raft_consensus.cc:348] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.868961 18987 raft_consensus.cc:374] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:35.869127 18987 raft_consensus.cc:729] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40689cb6f0814e2cae383be36c175beb, State: Initialized, Role: FOLLOWER
I20260926 23:16:35.869585 18987 consensus_queue.cc:260] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.870105 18986 tablet_bootstrap.cc:492] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: No bootstrap required, opened a new log
I20260926 23:16:35.870404 18986 ts_tablet_manager.cc:1397] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260926 23:16:35.870564 18987 ts_tablet_manager.cc:1428] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:35.870811 18985 tablet_bootstrap.cc:654] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:35.871378 18986 raft_consensus.cc:348] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.871696 18986 raft_consensus.cc:374] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:35.871860 18986 raft_consensus.cc:729] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056163913b27480b830b03784577d190, State: Initialized, Role: FOLLOWER
I20260926 23:16:35.872258 18986 consensus_queue.cc:260] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.872987 18986 ts_tablet_manager.cc:1428] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 23:16:35.875484 18985 tablet_bootstrap.cc:492] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: No bootstrap required, opened a new log
I20260926 23:16:35.875922 18985 ts_tablet_manager.cc:1397] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260926 23:16:35.877378 18985 raft_consensus.cc:348] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.877771 18985 raft_consensus.cc:374] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:35.877851 18985 raft_consensus.cc:729] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5f3ec0282945cfbe01b1f3cc059e7f, State: Initialized, Role: FOLLOWER
I20260926 23:16:35.878095 18985 consensus_queue.cc:260] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.878575 18985 ts_tablet_manager.cc:1428] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260926 23:16:35.902590 18995 raft_consensus.cc:482] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:35.902850 18995 raft_consensus.cc:504] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.904313 18548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78bbe6a7aca4ee9a15edb5d9553d43d" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056163913b27480b830b03784577d190" is_pre_election: true
I20260926 23:16:35.904655 18548 raft_consensus.cc:2454] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 0.
I20260926 23:16:35.905418 18618 leader_election.cc:302] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056163913b27480b830b03784577d190, cd5f3ec0282945cfbe01b1f3cc059e7f; no voters:
I20260926 23:16:35.905901 18957 raft_consensus.cc:2781] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:35.906085 18957 raft_consensus.cc:482] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:35.906458 18957 raft_consensus.cc:3037] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:35.912930 18995 leader_election.cc:288] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 056163913b27480b830b03784577d190 (127.16.62.65:36969), 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275)
I20260926 23:16:35.911386 18957 raft_consensus.cc:504] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.913053 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78bbe6a7aca4ee9a15edb5d9553d43d" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb" is_pre_election: true
I20260926 23:16:35.913455 18828 raft_consensus.cc:2454] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 0.
I20260926 23:16:35.914600 18548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78bbe6a7aca4ee9a15edb5d9553d43d" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056163913b27480b830b03784577d190"
I20260926 23:16:35.914898 18548 raft_consensus.cc:3037] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:35.916574 18957 leader_election.cc:288] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 election: Requested vote from peers 056163913b27480b830b03784577d190 (127.16.62.65:36969), 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275)
I20260926 23:16:35.919101 18548 raft_consensus.cc:2454] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 1.
I20260926 23:16:35.919597 18618 leader_election.cc:302] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056163913b27480b830b03784577d190, cd5f3ec0282945cfbe01b1f3cc059e7f; no voters:
I20260926 23:16:35.919947 18957 raft_consensus.cc:2781] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:35.920014 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78bbe6a7aca4ee9a15edb5d9553d43d" candidate_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40689cb6f0814e2cae383be36c175beb"
I20260926 23:16:35.920253 18957 raft_consensus.cc:686] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 LEADER]: Becoming Leader. State: Replica: cd5f3ec0282945cfbe01b1f3cc059e7f, State: Running, Role: LEADER
I20260926 23:16:35.920305 18828 raft_consensus.cc:3037] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:35.920699 18957 consensus_queue.cc:237] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } }
I20260926 23:16:35.924468 18828 raft_consensus.cc:2454] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5f3ec0282945cfbe01b1f3cc059e7f in term 1.
I20260926 23:16:35.925827 18387 catalog_manager.cc:5495] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f reported cstate change: term changed from 0 to 1, leader changed from <none> to cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66). New cstate: current_term: 1 leader_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd5f3ec0282945cfbe01b1f3cc059e7f" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 } health_report { overall_health: UNKNOWN } } }
I20260926 23:16:35.952365 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:35.947123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.16.62.66:36023)
0926 23:16:35.947777 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:35.947781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:35.947799 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:16:35.947856 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:35.947859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:35.947911 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:35.947987 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:35.948337 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:35.949384 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:35.950645 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:35.951104 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:35.951240 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:35.951587 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:35.951689 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:35.951758 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:35.951958 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 23:16:35.952020 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":418,"threads_started":1}
I20260926 23:16:35.960605 18548 raft_consensus.cc:1261] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Refusing update from remote peer cd5f3ec0282945cfbe01b1f3cc059e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:35.960611 18828 raft_consensus.cc:1261] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Refusing update from remote peer cd5f3ec0282945cfbe01b1f3cc059e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:35.961128 18962 consensus_queue.cc:1035] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40689cb6f0814e2cae383be36c175beb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:35.961393 18995 consensus_queue.cc:1035] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "056163913b27480b830b03784577d190" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:16:35.979646 18955 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 23:16:37.556041 18387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:37.556332 18387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:37.562006 18387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b0f894e568f4e3fb4a8e699e734b8bc: Sending DeleteTablet for 3 replicas of tablet a78bbe6a7aca4ee9a15edb5d9553d43d
I20260926 23:16:37.562990 18526 tablet_service.cc:1514] Processing DeleteTablet for tablet a78bbe6a7aca4ee9a15edb5d9553d43d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:37 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:43508
I20260926 23:16:37.562990 18664 tablet_service.cc:1514] Processing DeleteTablet for tablet a78bbe6a7aca4ee9a15edb5d9553d43d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:37 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:51876
I20260926 23:16:37.563042 18804 tablet_service.cc:1514] Processing DeleteTablet for tablet a78bbe6a7aca4ee9a15edb5d9553d43d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:37 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:46024
I20260926 23:16:37.563575 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 18460
I20260926 23:16:37.576177 19030 tablet_replica.cc:330] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: stopping tablet replica
I20260926 23:16:37.576692 19030 raft_consensus.cc:2229] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:37.577281 19030 raft_consensus.cc:2258] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:37.578822 19030 ts_tablet_manager.cc:1905] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:37.588476 19031 tablet_replica.cc:330] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: stopping tablet replica
I20260926 23:16:37.588792 19031 raft_consensus.cc:2229] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:37.589224 19031 raft_consensus.cc:2258] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:37.590456 19029 tablet_replica.cc:330] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: stopping tablet replica
I20260926 23:16:37.590479 19031 ts_tablet_manager.cc:1905] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:37.590617 19030 ts_tablet_manager.cc:1918] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:16:37.590777 19029 raft_consensus.cc:2229] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:16:37.590885 19030 log.cc:1196] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/a78bbe6a7aca4ee9a15edb5d9553d43d
I20260926 23:16:37.591354 19029 raft_consensus.cc:2258] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:37.592388 19030 ts_tablet_manager.cc:1939] T a78bbe6a7aca4ee9a15edb5d9553d43d P 056163913b27480b830b03784577d190: Deleting consensus metadata
I20260926 23:16:37.593681 19029 ts_tablet_manager.cc:1905] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:37.593652 18375 catalog_manager.cc:4841] TS 056163913b27480b830b03784577d190 (127.16.62.65:36969): tablet a78bbe6a7aca4ee9a15edb5d9553d43d (table test-table [id=591409bbe8a046658d0c34bda7a4709c]) successfully deleted
I20260926 23:16:37.596843 19031 ts_tablet_manager.cc:1918] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:16:37.596982 19031 log.cc:1196] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/a78bbe6a7aca4ee9a15edb5d9553d43d
I20260926 23:16:37.597402 19031 ts_tablet_manager.cc:1939] T a78bbe6a7aca4ee9a15edb5d9553d43d P 40689cb6f0814e2cae383be36c175beb: Deleting consensus metadata
I20260926 23:16:37.598297 18373 catalog_manager.cc:4841] TS 40689cb6f0814e2cae383be36c175beb (127.16.62.67:33275): tablet a78bbe6a7aca4ee9a15edb5d9553d43d (table test-table [id=591409bbe8a046658d0c34bda7a4709c]) successfully deleted
Sep 26 23:16:37 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:37.601441 19029 ts_tablet_manager.cc:1918] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:16:37.601629 19029 log.cc:1196] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/a78bbe6a7aca4ee9a15edb5d9553d43d
I20260926 23:16:37.602149 19029 ts_tablet_manager.cc:1939] T a78bbe6a7aca4ee9a15edb5d9553d43d P cd5f3ec0282945cfbe01b1f3cc059e7f: Deleting consensus metadata
I20260926 23:16:37.603420 18376 catalog_manager.cc:4841] TS cd5f3ec0282945cfbe01b1f3cc059e7f (127.16.62.66:36023): tablet a78bbe6a7aca4ee9a15edb5d9553d43d (table test-table [id=591409bbe8a046658d0c34bda7a4709c]) successfully deleted
I20260926 23:16:37.607573 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:37.564954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.16.62.65:36969)
0926 23:16:37.586830 (+ 21876us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:37.586838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:37.586868 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:16:37.586936 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:37.586944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:37.587042 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:37.587153 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:37.589819 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:37.592322 (+ 2503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:37.594500 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:37.595276 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:37.595389 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:37.601231 (+ 5842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:37.601254 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:37.601277 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:37.601328 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:37.605009 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:37.605569 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:37.605576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:37.605578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:37.605734 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:37.606158 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:37.606162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:37.606164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:37.606528 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:37.606768 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:37.607030 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 23:16:37.607189 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21706,"thread_start_us":21785,"threads_started":1}
W20260926 23:16:37.608331 18591 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43532
W20260926 23:16:37.609021 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 18460: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:16:37.609226 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 18460
I20260926 23:16:37.631119 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 18600
Sep 26 23:16:37 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:16:37.646574 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:37.632002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.16.62.66:36023)
0926 23:16:37.632465 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:37.632470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:37.632484 (+ 14us) server_negotiation.cc:407] Connection header received
0926 23:16:37.632530 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:37.632533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:37.632589 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:37.632670 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:37.634433 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:37.635144 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:37.636544 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:37.636941 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:37.637058 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:37.641498 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:37.641509 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:37.641523 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:37.641549 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:37.644006 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:37.644739 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:37.644745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:37.644747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:37.644871 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:37.645210 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:37.645213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:37.645215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:37.645523 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:37.645711 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:37.646017 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 23:16:37.646160 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
W20260926 23:16:37.647325 18730 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51924
W20260926 23:16:37.647887 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 18600: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:16:37.648025 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 18600
I20260926 23:16:37.673818 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 18739
Sep 26 23:16:37 dist-test-slave-vcv1 krb5kdc[18344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464595, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:16:37.689922 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:37.674547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.16.62.67:33275)
0926 23:16:37.675031 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:37.675035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:37.675050 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:16:37.675099 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:37.675103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:37.675156 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:37.675259 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:37.676661 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:37.677388 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:37.678699 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:37.679084 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:37.679172 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:37.683580 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:37.683594 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:37.683610 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:37.683640 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:37.686592 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:37.687362 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:37.687371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:37.687375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:37.687559 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:37.688009 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:37.688014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:37.688017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:37.688530 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:37.688824 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:37.689202 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 23:16:37.689436 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":268,"threads_started":1}
W20260926 23:16:37.690680 18869 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46040
W20260926 23:16:37.691397 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 18739: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:16:37.691560 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 18739
I20260926 23:16:37.711447 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 18357
I20260926 23:16:37.712642 18429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:37.829046 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 18357
2026-09-26T23:16:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8268 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:16:37 dist-test-slave-vcv1 krb5kdc[19039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:16:37 dist-test-slave-vcv1 krb5kdc[19039](info): set up 2 sockets
Sep 26 23:16:37 dist-test-slave-vcv1 krb5kdc[19039](info): commencing operation
krb5kdc: starting...
W20260926 23:16:38.933323 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:16:39 dist-test-slave-vcv1 krb5kdc[19039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:16:39.113204 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:36151
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:36151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:39.414950 19052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:39.415208 19052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:39.415371 19052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:39.422326 19052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:39.422420 19052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:39.422514 19052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:39.422608 19052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:39.422685 19052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:39.431911 19052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:36151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:39.433151 19052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:39.434417 19052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:39.445819 19060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:39.445883 19057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:39.445928 19058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:39.447383 19052 server_base.cc:1034] running on GCE node
I20260926 23:16:39.448040 19052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:39.449280 19052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:39.450466 19052 hybrid_clock.cc:648] HybridClock initialized: now 1790464599450419 us; error 49 us; skew 500 ppm
I20260926 23:16:39.450783 19052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:39 dist-test-slave-vcv1 krb5kdc[19039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464599, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:39.453743 19052 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:16:39.454970 19052 webserver.cc:460] Webserver started at http://127.16.62.126:44773/ using document root <none> and password file <none>
I20260926 23:16:39.455551 19052 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:39.455642 19052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:39.455910 19052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:39.458612 19052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "8bc15fb42ae54f0a88923bd4b256d352"
format_stamp: "Formatted at 2026-09-26 23:16:39 on dist-test-slave-vcv1"
server_key: "4e32437b1daf5972b6f80422f8beed92"
server_key_iv: "987474560de59b0aedf82cc3da829388"
server_key_version: "encryptionkey@0"
I20260926 23:16:39.459242 19052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "8bc15fb42ae54f0a88923bd4b256d352"
format_stamp: "Formatted at 2026-09-26 23:16:39 on dist-test-slave-vcv1"
server_key: "4e32437b1daf5972b6f80422f8beed92"
server_key_iv: "987474560de59b0aedf82cc3da829388"
server_key_version: "encryptionkey@0"
I20260926 23:16:39.463791 19052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 23:16:39.466923 19067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:39.468281 19052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:39.468459 19052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "8bc15fb42ae54f0a88923bd4b256d352"
format_stamp: "Formatted at 2026-09-26 23:16:39 on dist-test-slave-vcv1"
server_key: "4e32437b1daf5972b6f80422f8beed92"
server_key_iv: "987474560de59b0aedf82cc3da829388"
server_key_version: "encryptionkey@0"
I20260926 23:16:39.468633 19052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:39.498968 19052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:39.503275 19052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:39.503566 19052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:39.521993 19052 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:36151
I20260926 23:16:39.522017 19118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:36151 every 8 connection(s)
I20260926 23:16:39.523622 19052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:39.527313 19119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:39.528577 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 19052
I20260926 23:16:39.528784 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:39.529405 16633 external_mini_cluster.cc:1428] Setting key 64186951378573589cd22e08d294c7b8
I20260926 23:16:39.541198 19119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352: Bootstrap starting.
Sep 26 23:16:39 dist-test-slave-vcv1 krb5kdc[19039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:39.544765 19119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:39.545822 19119 log.cc:824] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:39.549088 19119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352: No bootstrap required, opened a new log
I20260926 23:16:39.552672 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:39.531238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.16.62.126:36151)
0926 23:16:39.531990 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:39.532003 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:39.532041 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:16:39.533621 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:39.533645 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:39.534020 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:39.534348 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:39.536255 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:39.537508 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:39.539318 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:39.539950 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:39.540077 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:39.544596 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:39.544603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:39.544616 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:39.544649 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:39.548261 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:39.548923 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:39.548931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:39.548939 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:39.549143 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:39.549519 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:39.549526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:39.549530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:39.550231 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:39.550511 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:39.550833 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 23:16:39.551274 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":346,"threads_started":1}
I20260926 23:16:39.558516 19119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc15fb42ae54f0a88923bd4b256d352" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36151 } }
I20260926 23:16:39.558830 19119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:39.558902 19119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc15fb42ae54f0a88923bd4b256d352, State: Initialized, Role: FOLLOWER
I20260926 23:16:39.559432 19119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc15fb42ae54f0a88923bd4b256d352" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36151 } }
I20260926 23:16:39.559818 19119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:39.559937 19119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:39.560053 19119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:39.562405 19119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc15fb42ae54f0a88923bd4b256d352" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36151 } }
I20260926 23:16:39.562821 19119 leader_election.cc:302] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bc15fb42ae54f0a88923bd4b256d352; no voters:
I20260926 23:16:39.563303 19119 leader_election.cc:288] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:39.563462 19124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:39.564239 19124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc15fb42ae54f0a88923bd4b256d352, State: Running, Role: LEADER
I20260926 23:16:39.564776 19124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc15fb42ae54f0a88923bd4b256d352" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36151 } }
I20260926 23:16:39.565690 19119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:39.567799 19126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc15fb42ae54f0a88923bd4b256d352. Latest consensus state: current_term: 1 leader_uuid: "8bc15fb42ae54f0a88923bd4b256d352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc15fb42ae54f0a88923bd4b256d352" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36151 } } }
I20260926 23:16:39.568167 19125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bc15fb42ae54f0a88923bd4b256d352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc15fb42ae54f0a88923bd4b256d352" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36151 } } }
I20260926 23:16:39.568353 19126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:39.568441 19125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:39.572587 19132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:39.576829 19132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:39.585391 19132 catalog_manager.cc:1347] Generated new cluster ID: 85748192436b4a479663d3dd345553bf
I20260926 23:16:39.585501 19132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:39.604588 19132 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:39.605789 19132 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:39.614190 19132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bc15fb42ae54f0a88923bd4b256d352: Generated new TSK 0
I20260926 23:16:39.615057 19132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:16:39.663623 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36151
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:39.983490 19145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:39.983798 19145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:39.983969 19145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:39.991183 19145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:39.991322 19145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:39.991504 19145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:16:40.000910 19145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:40.002156 19145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:40.003464 19145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:40.016381 19150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:40.016351 19153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:40.016433 19151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:40.017346 19145 server_base.cc:1034] running on GCE node
I20260926 23:16:40.017954 19145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:40.018894 19145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:40.020187 19145 hybrid_clock.cc:648] HybridClock initialized: now 1790464600020101 us; error 50 us; skew 500 ppm
I20260926 23:16:40.020505 19145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:40 dist-test-slave-vcv1 krb5kdc[19039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:40.023613 19145 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:16:40.025125 19145 webserver.cc:460] Webserver started at http://127.16.62.65:36879/ using document root <none> and password file <none>
I20260926 23:16:40.025781 19145 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:40.025919 19145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:40.026222 19145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:40.029140 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "8f35d8ba0e184bfbbdbf643e562e882d"
format_stamp: "Formatted at 2026-09-26 23:16:40 on dist-test-slave-vcv1"
server_key: "abf601842b22ccc97c49a039d25f0fe5"
server_key_iv: "86705360cf2d42a4e1e22b768bd49ef2"
server_key_version: "encryptionkey@0"
I20260926 23:16:40.029909 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "8f35d8ba0e184bfbbdbf643e562e882d"
format_stamp: "Formatted at 2026-09-26 23:16:40 on dist-test-slave-vcv1"
server_key: "abf601842b22ccc97c49a039d25f0fe5"
server_key_iv: "86705360cf2d42a4e1e22b768bd49ef2"
server_key_version: "encryptionkey@0"
I20260926 23:16:40.034878 19145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 23:16:40.038175 19160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:40.039489 19145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:16:40.039669 19145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "8f35d8ba0e184bfbbdbf643e562e882d"
format_stamp: "Formatted at 2026-09-26 23:16:40 on dist-test-slave-vcv1"
server_key: "abf601842b22ccc97c49a039d25f0fe5"
server_key_iv: "86705360cf2d42a4e1e22b768bd49ef2"
server_key_version: "encryptionkey@0"
I20260926 23:16:40.039862 19145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:40.054081 19145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:40.058195 19145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:40.058473 19145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:40.059374 19145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:40.060696 19145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:40.060817 19145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:40.060916 19145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:40.060966 19145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:40.095079 19145 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:45567
I20260926 23:16:40.095110 19272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:45567 every 8 connection(s)
I20260926 23:16:40.096511 19145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:16:40.101740 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 19145
I20260926 23:16:40.101939 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:16:40.102571 16633 external_mini_cluster.cc:1428] Setting key 81dc2bae0108e6e356638a13f87525cf
Sep 26 23:16:40 dist-test-slave-vcv1 krb5kdc[19039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:40.117817 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:40.099024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:50205 (local address 127.16.62.126:36151)
0926 23:16:40.099452 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:40.099457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:40.100343 (+ 886us) server_negotiation.cc:407] Connection header received
0926 23:16:40.102219 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:40.102225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:40.102321 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:40.102437 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:40.104239 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:40.105343 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:40.106948 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:40.107675 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:40.107816 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:40.111239 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:40.111248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:40.111257 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:40.111298 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:40.114564 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:40.115266 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:40.115271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:40.115274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:40.115390 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:40.115833 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:40.115838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:40.115841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:40.116109 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:40.116362 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:40.117109 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 23:16:40.117344 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":238,"threads_started":1}
I20260926 23:16:40.118436 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:40.099475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36151 (local address 127.16.62.65:50205)
0926 23:16:40.100131 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:40.100216 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:40.101925 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:40.102702 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:40.102712 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:40.103300 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:40.104036 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:40.104058 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:40.105499 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:40.105503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:40.106786 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:40.106799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:40.107875 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:40.107880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:40.108073 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:40.108776 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:40.108797 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:40.111032 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:40.114715 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:40.114730 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:40.114743 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:40.115131 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:40.115512 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:40.115516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:40.115518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:40.115701 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:40.116329 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:40.116337 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:40.116679 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 23:16:40.116962 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 23:16:40.117356 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":255,"threads_started":1}
I20260926 23:16:40.120009 19273 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36151
I20260926 23:16:40.120371 19273 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
I20260926 23:16:40.121158 19273 heartbeater.cc:511] Master 127.16.62.126:36151 requested a full tablet report, sending...
I20260926 23:16:40.123219 19084 ts_manager.cc:194] Registered new tserver with Master: 8f35d8ba0e184bfbbdbf643e562e882d (127.16.62.65:45567)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
I20260926 23:16:40.125371 19084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:50205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:16:40.147116 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36151
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:40.457738 19279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:40.458045 19279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:40.458221 19279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:40.465181 19279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:40.465322 19279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:40.465485 19279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:16:40.474792 19279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:40.475919 19279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:40.477136 19279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:40.490515 19284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:40.490515 19287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:40.490552 19285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:40.491456 19279 server_base.cc:1034] running on GCE node
I20260926 23:16:40.492034 19279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:40.492952 19279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:40.494174 19279 hybrid_clock.cc:648] HybridClock initialized: now 1790464600494133 us; error 58 us; skew 500 ppm
I20260926 23:16:40.494472 19279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:40 dist-test-slave-vcv1 krb5kdc[19039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:40.497447 19279 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:16:40.498842 19279 webserver.cc:460] Webserver started at http://127.16.62.66:38735/ using document root <none> and password file <none>
I20260926 23:16:40.499501 19279 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:40.499651 19279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:40.499943 19279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:40.502730 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "6db19e2137e44c38945612784fd3c5df"
format_stamp: "Formatted at 2026-09-26 23:16:40 on dist-test-slave-vcv1"
server_key: "c4917e2a1f4890599f24bf9b3e009b53"
server_key_iv: "9d89c7f5a15e56c57846fc66f22658b6"
server_key_version: "encryptionkey@0"
I20260926 23:16:40.503496 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "6db19e2137e44c38945612784fd3c5df"
format_stamp: "Formatted at 2026-09-26 23:16:40 on dist-test-slave-vcv1"
server_key: "c4917e2a1f4890599f24bf9b3e009b53"
server_key_iv: "9d89c7f5a15e56c57846fc66f22658b6"
server_key_version: "encryptionkey@0"
I20260926 23:16:40.508584 19279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 23:16:40.511752 19294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:40.513020 19279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:40.513182 19279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "6db19e2137e44c38945612784fd3c5df"
format_stamp: "Formatted at 2026-09-26 23:16:40 on dist-test-slave-vcv1"
server_key: "c4917e2a1f4890599f24bf9b3e009b53"
server_key_iv: "9d89c7f5a15e56c57846fc66f22658b6"
server_key_version: "encryptionkey@0"
I20260926 23:16:40.513360 19279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:40.534302 19279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:40.538342 19279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:40.538622 19279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:40.539556 19279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:40.540899 19279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:40.541002 19279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:40.541142 19279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:40.541211 19279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:40.574275 19279 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:40045
I20260926 23:16:40.574327 19406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:40045 every 8 connection(s)
I20260926 23:16:40.575500 19279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:16:40.583943 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 19279
I20260926 23:16:40.584103 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:16:40.584671 16633 external_mini_cluster.cc:1428] Setting key eebb54003562ba73b50e95b1142ab179
Sep 26 23:16:40 dist-test-slave-vcv1 krb5kdc[19039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:40.595709 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:40.578010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:54503 (local address 127.16.62.126:36151)
0926 23:16:40.578181 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:40.578185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:40.579199 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 23:16:40.580980 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:40.580984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:40.581047 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:40.581131 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:40.582516 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:40.583218 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:40.585366 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:40.585815 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:40.585907 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:40.589601 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:40.589610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:40.589618 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:40.589679 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:40.592749 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:40.593646 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:40.593650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:40.593653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:40.593772 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:40.594229 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:40.594232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:40.594234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:40.594465 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:40.594561 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:40.595345 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 23:16:40.595490 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:16:40.597311 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:40.578392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36151 (local address 127.16.62.66:54503)
0926 23:16:40.579016 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:40.579056 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:40.580749 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:40.581330 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:40.581337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:40.581795 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:40.582381 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:40.582397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:40.583339 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:40.583342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:40.585243 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:40.585253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:40.586020 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:40.586027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:40.586278 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:40.587000 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:40.587025 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:40.589376 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:40.592882 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:40.592891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:40.592906 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:40.593520 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:40.593874 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:40.593882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:40.593888 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:40.594119 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:40.594585 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:40.594594 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:40.594996 (+ 402us) client_negotiation.cc:769] Sending connection context
0926 23:16:40.595278 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 23:16:40.595856 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":245,"threads_started":1}
I20260926 23:16:40.598685 19407 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36151
I20260926 23:16:40.598961 19407 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:40.599602 19407 heartbeater.cc:511] Master 127.16.62.126:36151 requested a full tablet report, sending...
I20260926 23:16:40.601025 19084 ts_manager.cc:194] Registered new tserver with Master: 6db19e2137e44c38945612784fd3c5df (127.16.62.66:40045)
I20260926 23:16:40.601828 19084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:54503
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:16:40.627938 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36151
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:40.956419 19412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:40.956717 19412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:40.956938 19412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:40.964064 19412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:40.964236 19412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:40.964404 19412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:16:40.974802 19412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:40.976058 19412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:40.977401 19412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:40.990217 19417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:40.990207 19418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:40.990460 19420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:40.991171 19412 server_base.cc:1034] running on GCE node
I20260926 23:16:40.991779 19412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:40.992843 19412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:40.994031 19412 hybrid_clock.cc:648] HybridClock initialized: now 1790464600993992 us; error 36 us; skew 500 ppm
I20260926 23:16:40.994347 19412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:40 dist-test-slave-vcv1 krb5kdc[19039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:40.997399 19412 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:16:40.998718 19412 webserver.cc:460] Webserver started at http://127.16.62.67:39155/ using document root <none> and password file <none>
I20260926 23:16:40.999356 19412 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:40.999492 19412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:40.999783 19412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:41.002574 19412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "6459dbb554f04dcfa1b0aa70556fca9d"
format_stamp: "Formatted at 2026-09-26 23:16:40 on dist-test-slave-vcv1"
server_key: "2ed4fc79bdb8348757fdcaf48ee4ca4d"
server_key_iv: "e7c4c16d68802816cea5e78e4ad412b1"
server_key_version: "encryptionkey@0"
I20260926 23:16:41.003404 19412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "6459dbb554f04dcfa1b0aa70556fca9d"
format_stamp: "Formatted at 2026-09-26 23:16:40 on dist-test-slave-vcv1"
server_key: "2ed4fc79bdb8348757fdcaf48ee4ca4d"
server_key_iv: "e7c4c16d68802816cea5e78e4ad412b1"
server_key_version: "encryptionkey@0"
I20260926 23:16:41.008208 19412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 23:16:41.011345 19427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:41.012655 19412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:41.012816 19412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "6459dbb554f04dcfa1b0aa70556fca9d"
format_stamp: "Formatted at 2026-09-26 23:16:40 on dist-test-slave-vcv1"
server_key: "2ed4fc79bdb8348757fdcaf48ee4ca4d"
server_key_iv: "e7c4c16d68802816cea5e78e4ad412b1"
server_key_version: "encryptionkey@0"
I20260926 23:16:41.013008 19412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:41.028043 19412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:41.032248 19412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:41.032503 19412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:41.033344 19412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:41.034508 19412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:41.034610 19412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:41.034749 19412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:41.034822 19412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:41.066593 19412 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:41319
I20260926 23:16:41.066613 19539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:41319 every 8 connection(s)
I20260926 23:16:41.067852 19412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:16:41.073997 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 19412
I20260926 23:16:41.074167 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:16:41.074604 16633 external_mini_cluster.cc:1428] Setting key 04fed65397921ead7dd7e0dea4cee067
Sep 26 23:16:41 dist-test-slave-vcv1 krb5kdc[19039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464600, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:41.087016 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:41.070360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:46089 (local address 127.16.62.126:36151)
0926 23:16:41.070541 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:41.070546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:41.071703 (+ 1157us) server_negotiation.cc:407] Connection header received
0926 23:16:41.073688 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:41.073694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:41.073785 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:41.073892 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:41.075535 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:41.076527 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:41.078113 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:41.078497 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:41.078668 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:41.081874 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:41.081888 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:41.081891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:41.081920 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:41.084245 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:41.084926 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:41.084933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:41.084937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:41.085058 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:41.085508 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:41.085515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:41.085518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:41.085772 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:41.085907 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:41.086485 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 23:16:41.086678 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:16:41.087915 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:41.070819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36151 (local address 127.16.62.67:46089)
0926 23:16:41.071539 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:41.071600 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:41.073403 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:41.074121 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:41.074152 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:41.074677 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:41.075376 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:41.075395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:41.076667 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:41.076673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:41.077975 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:41.077987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:41.078739 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:41.078749 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:41.078992 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:41.079577 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:41.079597 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:41.081680 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:41.084415 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:41.084423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:41.084437 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:41.084810 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:41.085181 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:41.085185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:41.085189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:41.085405 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:41.085883 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:41.085890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:41.086159 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 23:16:41.086414 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 23:16:41.086868 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":276,"threads_started":1}
I20260926 23:16:41.089324 19540 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36151
I20260926 23:16:41.089623 19540 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:41.090234 19540 heartbeater.cc:511] Master 127.16.62.126:36151 requested a full tablet report, sending...
I20260926 23:16:41.091615 19084 ts_manager.cc:194] Registered new tserver with Master: 6459dbb554f04dcfa1b0aa70556fca9d (127.16.62.67:41319)
I20260926 23:16:41.092382 19084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:46089
I20260926 23:16:41.101512 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:16:41.128266 19273 heartbeater.cc:503] Master 127.16.62.126:36151 was elected leader, sending a full tablet report...
W20260926 23:16:41.131769 19275 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:16:41.129107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.16.62.126:36151)
0926 23:16:41.129393 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:41.129415 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:41.129898 (+ 483us) server_negotiation.cc:407] Connection header received
0926 23:16:41.130117 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:41.130121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:41.130184 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:41.130266 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:41.131483 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41366: BlockingRecv error: recv got EOF from 127.0.0.1:41366 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:16:41.134904 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19145
W20260926 23:16:41.137470 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19145: Not authorized: Client connection negotiation failed: client connection to 127.16.62.65:45567: server requires authentication, but client does not have Kerberos credentials available
I20260926 23:16:41.137666 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 19145
I20260926 23:16:41.149905 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19279
W20260926 23:16:41.152360 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19279: Not authorized: Client connection negotiation failed: client connection to 127.16.62.66:40045: server requires authentication, but client does not have Kerberos credentials available
I20260926 23:16:41.152493 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 19279
I20260926 23:16:41.167690 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19412
W20260926 23:16:41.170346 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19412: Not authorized: Client connection negotiation failed: client connection to 127.16.62.67:41319: server requires authentication, but client does not have Kerberos credentials available
I20260926 23:16:41.170481 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 19412
I20260926 23:16:41.185822 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19052
I20260926 23:16:41.187013 19114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:41.287660 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 19052
2026-09-26T23:16:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3456 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:16:41 dist-test-slave-vcv1 krb5kdc[19557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:16:41 dist-test-slave-vcv1 krb5kdc[19557](info): set up 2 sockets
Sep 26 23:16:41 dist-test-slave-vcv1 krb5kdc[19557](info): commencing operation
krb5kdc: starting...
W20260926 23:16:42.361413 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:16:42 dist-test-slave-vcv1 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:16:42.540421 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44549
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:36883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:44549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:42.847633 19570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:42.847887 19570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:42.847971 19570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:42.854926 19570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:42.855011 19570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:42.855113 19570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:42.855212 19570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:42.855309 19570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:42.864527 19570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:36883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:44549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:42.865726 19570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:42.866937 19570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:42.878171 19575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:42.878263 19576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:42.878222 19578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:42.879369 19570 server_base.cc:1034] running on GCE node
I20260926 23:16:42.880260 19570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:42.881592 19570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:42.882812 19570 hybrid_clock.cc:648] HybridClock initialized: now 1790464602882778 us; error 30 us; skew 500 ppm
I20260926 23:16:42.883126 19570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:42 dist-test-slave-vcv1 krb5kdc[19557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464602, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:42.886173 19570 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:16:42.887604 19570 webserver.cc:460] Webserver started at http://127.16.62.126:45527/ using document root <none> and password file <none>
I20260926 23:16:42.888329 19570 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:42.888464 19570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:42.888797 19570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:42.891706 19570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b"
format_stamp: "Formatted at 2026-09-26 23:16:42 on dist-test-slave-vcv1"
server_key: "63b2b4d25465ea8df8e67d59cbea9078"
server_key_iv: "9b42b5b5952b15f6d93447e703dea07d"
server_key_version: "encryptionkey@0"
I20260926 23:16:42.892506 19570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b"
format_stamp: "Formatted at 2026-09-26 23:16:42 on dist-test-slave-vcv1"
server_key: "63b2b4d25465ea8df8e67d59cbea9078"
server_key_iv: "9b42b5b5952b15f6d93447e703dea07d"
server_key_version: "encryptionkey@0"
I20260926 23:16:42.897631 19570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:16:42.900874 19585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:42.902102 19570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:42.902287 19570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b"
format_stamp: "Formatted at 2026-09-26 23:16:42 on dist-test-slave-vcv1"
server_key: "63b2b4d25465ea8df8e67d59cbea9078"
server_key_iv: "9b42b5b5952b15f6d93447e703dea07d"
server_key_version: "encryptionkey@0"
I20260926 23:16:42.902472 19570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:42.929174 19570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:42.933076 19570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:42.933364 19570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:42.951287 19570 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:44549
I20260926 23:16:42.951316 19636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:44549 every 8 connection(s)
I20260926 23:16:42.952649 19570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:42.956204 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 19570
I20260926 23:16:42.956248 19637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:42.956454 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:42.957234 16633 external_mini_cluster.cc:1428] Setting key 49989ef87e4fc0a7d2cc5773e1c0ba52
I20260926 23:16:42.968327 19637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b: Bootstrap starting.
Sep 26 23:16:42 dist-test-slave-vcv1 krb5kdc[19557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:42.971450 19637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:42.972667 19637 log.cc:824] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:42.975137 19637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b: No bootstrap required, opened a new log
I20260926 23:16:42.977766 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:42.959063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.16.62.126:44549)
0926 23:16:42.959881 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:42.959891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:42.959926 (+ 35us) server_negotiation.cc:407] Connection header received
0926 23:16:42.961455 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:42.961473 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:42.961828 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:42.962202 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:42.963587 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:42.964898 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:42.966124 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:42.966821 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:42.967004 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:42.971083 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:42.971090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:42.971103 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:42.971138 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:42.974079 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:42.974695 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:42.974701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:42.974706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:42.974851 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:42.975207 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:42.975213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:42.975216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:42.975717 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:42.976019 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:42.976468 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 23:16:42.976859 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":378,"threads_started":1}
I20260926 23:16:42.981309 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:16:42.983026 19637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44549 } }
I20260926 23:16:42.983352 19637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:42.983451 19637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5b9a612d9c4de1b303d5ecc3ebad9b, State: Initialized, Role: FOLLOWER
I20260926 23:16:42.983968 19637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44549 } }
I20260926 23:16:42.984220 19637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:42.984361 19637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:42.984521 19637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:42.986519 19637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44549 } }
I20260926 23:16:42.986986 19637 leader_election.cc:302] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f5b9a612d9c4de1b303d5ecc3ebad9b; no voters:
I20260926 23:16:42.987602 19637 leader_election.cc:288] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:42.987705 19643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:42.989945 19637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:42.994274 19643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [term 1 LEADER]: Becoming Leader. State: Replica: 5f5b9a612d9c4de1b303d5ecc3ebad9b, State: Running, Role: LEADER
I20260926 23:16:42.994984 19643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44549 } }
I20260926 23:16:42.999742 19644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44549 } } }
I20260926 23:16:43.000166 19644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:43.000797 19645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f5b9a612d9c4de1b303d5ecc3ebad9b. Latest consensus state: current_term: 1 leader_uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b9a612d9c4de1b303d5ecc3ebad9b" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44549 } } }
I20260926 23:16:43.001149 19645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:43.010007 19655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:43.014564 19655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:43.023478 19655 catalog_manager.cc:1347] Generated new cluster ID: e8d2722e525c43779b68441d9cc6ea88
I20260926 23:16:43.023614 19655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:43.038149 19664 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:16:43.048182 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:43.040707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.16.62.126:44549)
0926 23:16:43.040910 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:43.040919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:43.041507 (+ 588us) server_negotiation.cc:407] Connection header received
0926 23:16:43.041717 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:43.041722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:43.041812 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:43.041920 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:43.042943 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:43.043643 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:43.044955 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:43.045382 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:43.045519 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:43.046557 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:43.046563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:43.046571 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:43.047352 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:43.047395 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:43.047421 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:43.047537 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:43.047701 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 23:16:43.047787 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:16:43.054085 19655 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:43.055300 19655 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:43.063150 19655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f5b9a612d9c4de1b303d5ecc3ebad9b: Generated new TSK 0
I20260926 23:16:43.064157 19655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:16:43.075739 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19570
I20260926 23:16:43.076975 19632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:43.164901 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 19570
2026-09-26T23:16:43Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1868 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:16:43 dist-test-slave-vcv1 krb5kdc[19671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:16:43 dist-test-slave-vcv1 krb5kdc[19671](info): set up 2 sockets
Sep 26 23:16:43 dist-test-slave-vcv1 krb5kdc[19671](info): commencing operation
krb5kdc: starting...
W20260926 23:16:44.229382 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:16:44 dist-test-slave-vcv1 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:16:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:16:44.419409 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:40793
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:40793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:44.729601 19684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:44.729882 19684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:44.730068 19684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:44.737277 19684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:44.737385 19684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:44.737514 19684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:44.737604 19684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:44.737672 19684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:44.747417 19684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:40793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:44.748823 19684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:44.750095 19684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:44.761572 19690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:44.761610 19689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:44.761646 19692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:44.762398 19684 server_base.cc:1034] running on GCE node
I20260926 23:16:44.763172 19684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:44.764575 19684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:44.765791 19684 hybrid_clock.cc:648] HybridClock initialized: now 1790464604765743 us; error 41 us; skew 500 ppm
I20260926 23:16:44.766120 19684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:44 dist-test-slave-vcv1 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464604, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:44.769161 19684 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:16:44.770606 19684 webserver.cc:460] Webserver started at http://127.16.62.126:45315/ using document root <none> and password file <none>
I20260926 23:16:44.771265 19684 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:44.771402 19684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:44.771696 19684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:44.774641 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "6c8ded65b22446f48244acbc92b664b8"
format_stamp: "Formatted at 2026-09-26 23:16:44 on dist-test-slave-vcv1"
server_key: "04a98b3dcac89622b687c645a182942c"
server_key_iv: "dedd126e2a520414d0880acf12e7ff14"
server_key_version: "encryptionkey@0"
I20260926 23:16:44.775391 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "6c8ded65b22446f48244acbc92b664b8"
format_stamp: "Formatted at 2026-09-26 23:16:44 on dist-test-slave-vcv1"
server_key: "04a98b3dcac89622b687c645a182942c"
server_key_iv: "dedd126e2a520414d0880acf12e7ff14"
server_key_version: "encryptionkey@0"
I20260926 23:16:44.780364 19684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 23:16:44.783564 19699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:44.784897 19684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:44.785080 19684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "6c8ded65b22446f48244acbc92b664b8"
format_stamp: "Formatted at 2026-09-26 23:16:44 on dist-test-slave-vcv1"
server_key: "04a98b3dcac89622b687c645a182942c"
server_key_iv: "dedd126e2a520414d0880acf12e7ff14"
server_key_version: "encryptionkey@0"
I20260926 23:16:44.785270 19684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:44.798167 19684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:44.802210 19684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:44.802488 19684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:44.820430 19684 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:40793
I20260926 23:16:44.820451 19750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:40793 every 8 connection(s)
I20260926 23:16:44.821867 19684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:44.825263 19751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:44.826237 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 19684
I20260926 23:16:44.826491 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:44.827294 16633 external_mini_cluster.cc:1428] Setting key 2e83a117e0e2bc089cadec6f8ba8be06
I20260926 23:16:44.839134 19751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8: Bootstrap starting.
Sep 26 23:16:44 dist-test-slave-vcv1 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:44.842572 19751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:44.843791 19751 log.cc:824] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:44.846531 19751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8: No bootstrap required, opened a new log
I20260926 23:16:44.849242 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:44.829338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.16.62.126:40793)
0926 23:16:44.830075 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:44.830086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:44.830127 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:16:44.831696 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:44.831715 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:44.832086 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:44.832471 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:44.834148 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:44.835402 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:44.837160 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:44.837775 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:44.837893 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:44.842340 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:44.842347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:44.842360 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:44.842391 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:44.845399 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:44.846135 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:44.846141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:44.846146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:44.846304 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:44.846648 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:44.846656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:44.846661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:44.847147 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:44.847476 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:44.847884 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 23:16:44.848323 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":342,"threads_started":1}
I20260926 23:16:44.854296 19751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8ded65b22446f48244acbc92b664b8" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40793 } }
I20260926 23:16:44.854633 19751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:44.854733 19751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8ded65b22446f48244acbc92b664b8, State: Initialized, Role: FOLLOWER
I20260926 23:16:44.855226 19751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8ded65b22446f48244acbc92b664b8" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40793 } }
I20260926 23:16:44.855445 19751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:44.855580 19751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:44.855739 19751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:44.858191 19751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8ded65b22446f48244acbc92b664b8" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40793 } }
I20260926 23:16:44.858630 19751 leader_election.cc:302] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c8ded65b22446f48244acbc92b664b8; no voters:
I20260926 23:16:44.859174 19751 leader_election.cc:288] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:44.859273 19756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:44.860010 19756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c8ded65b22446f48244acbc92b664b8, State: Running, Role: LEADER
I20260926 23:16:44.860503 19756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8ded65b22446f48244acbc92b664b8" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40793 } }
I20260926 23:16:44.861196 19751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:44.863384 19758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c8ded65b22446f48244acbc92b664b8. Latest consensus state: current_term: 1 leader_uuid: "6c8ded65b22446f48244acbc92b664b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8ded65b22446f48244acbc92b664b8" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40793 } } }
I20260926 23:16:44.863708 19758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:44.866055 19757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c8ded65b22446f48244acbc92b664b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8ded65b22446f48244acbc92b664b8" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40793 } } }
I20260926 23:16:44.866501 19757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:44.867301 19762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:44.873410 19762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:44.881582 19762 catalog_manager.cc:1347] Generated new cluster ID: 33da6b0b47ee4f5e9bdacb86f0a42f74
I20260926 23:16:44.881693 19762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:44.903539 19762 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:44.904919 19762 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:44.913529 19762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c8ded65b22446f48244acbc92b664b8: Generated new TSK 0
I20260926 23:16:44.914304 19762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:16:44.955819 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40793
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:45.270919 19777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:45.271250 19777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:45.271436 19777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:45.278558 19777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:45.278680 19777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:45.278859 19777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:16:45.288594 19777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:45.289927 19777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:45.291206 19777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:45.304330 19783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:45.304280 19785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:45.304427 19782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:45.304986 19777 server_base.cc:1034] running on GCE node
I20260926 23:16:45.305560 19777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:45.306385 19777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:45.307569 19777 hybrid_clock.cc:648] HybridClock initialized: now 1790464605307514 us; error 48 us; skew 500 ppm
I20260926 23:16:45.307855 19777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:45 dist-test-slave-vcv1 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464605, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:45.310875 19777 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:16:45.312273 19777 webserver.cc:460] Webserver started at http://127.16.62.65:44041/ using document root <none> and password file <none>
I20260926 23:16:45.312896 19777 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:45.313036 19777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:45.313326 19777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:45.316041 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "c569a2757b15466383fc96b5964de852"
format_stamp: "Formatted at 2026-09-26 23:16:45 on dist-test-slave-vcv1"
server_key: "d6f9a617617a161243ed3cc76b5ce8a1"
server_key_iv: "1f8080e635f5583f4134724ff98c650b"
server_key_version: "encryptionkey@0"
I20260926 23:16:45.316910 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "c569a2757b15466383fc96b5964de852"
format_stamp: "Formatted at 2026-09-26 23:16:45 on dist-test-slave-vcv1"
server_key: "d6f9a617617a161243ed3cc76b5ce8a1"
server_key_iv: "1f8080e635f5583f4134724ff98c650b"
server_key_version: "encryptionkey@0"
I20260926 23:16:45.321692 19777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:16:45.324937 19792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:45.326156 19777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:45.326318 19777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "c569a2757b15466383fc96b5964de852"
format_stamp: "Formatted at 2026-09-26 23:16:45 on dist-test-slave-vcv1"
server_key: "d6f9a617617a161243ed3cc76b5ce8a1"
server_key_iv: "1f8080e635f5583f4134724ff98c650b"
server_key_version: "encryptionkey@0"
I20260926 23:16:45.326498 19777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:45.345638 19777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:45.349823 19777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:45.350106 19777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:45.351095 19777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:45.352387 19777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:45.352489 19777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:45.352627 19777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:45.352700 19777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:45.385403 19777 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:33887
I20260926 23:16:45.385432 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:33887 every 8 connection(s)
I20260926 23:16:45.386770 19777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:16:45.393393 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 19777
I20260926 23:16:45.393615 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:16:45.394397 16633 external_mini_cluster.cc:1428] Setting key fcd38c3d4b503c3869c716ed4176c28b
Sep 26 23:16:45 dist-test-slave-vcv1 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464605, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:45.407874 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:45.389247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:59323 (local address 127.16.62.126:40793)
0926 23:16:45.389755 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:45.389763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:45.390768 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 23:16:45.392503 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:45.392508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:45.392586 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:45.392690 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:45.394619 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:45.395585 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:45.397879 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:45.398278 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:45.398375 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:45.402433 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:45.402438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:45.402441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:45.402470 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:45.405002 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:45.405729 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:45.405743 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:45.405746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:45.405851 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:45.406257 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:45.406262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:45.406265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:45.406525 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:45.406677 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:45.407303 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 23:16:45.407511 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":264,"threads_started":1}
I20260926 23:16:45.408943 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:45.389910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:40793 (local address 127.16.62.65:59323)
0926 23:16:45.390603 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:45.390646 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:45.392241 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:45.392877 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:45.392884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:45.393628 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:45.394413 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:45.394440 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:45.395739 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:45.395745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:45.397711 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:45.397729 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:45.398502 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:45.398512 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:45.398851 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:45.399567 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:45.399603 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:45.402213 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:45.405140 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:45.405148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:45.405163 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:45.405603 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:45.405960 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:45.405965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:45.405969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:45.406143 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:45.406648 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:45.406655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:45.406968 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 23:16:45.407216 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 23:16:45.407714 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":278,"threads_started":1}
I20260926 23:16:45.410423 19905 heartbeater.cc:347] Connected to a master server at 127.16.62.126:40793
I20260926 23:16:45.410724 19905 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:45.411409 19905 heartbeater.cc:511] Master 127.16.62.126:40793 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
I20260926 23:16:45.413610 19716 ts_manager.cc:194] Registered new tserver with Master: c569a2757b15466383fc96b5964de852 (127.16.62.65:33887)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
I20260926 23:16:45.415380 19716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:59323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:16:45.437288 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40793
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:45.758972 19911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:45.759250 19911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:45.759344 19911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:45.766711 19911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:45.766858 19911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:45.767031 19911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:16:45.777747 19911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:45.779078 19911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:45.780551 19911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:45.794842 19916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:45.794874 19917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:45.794915 19919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:45.795898 19911 server_base.cc:1034] running on GCE node
I20260926 23:16:45.796693 19911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:45.797698 19911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:45.798900 19911 hybrid_clock.cc:648] HybridClock initialized: now 1790464605798851 us; error 35 us; skew 500 ppm
I20260926 23:16:45.799249 19911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:45 dist-test-slave-vcv1 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464605, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:45.802675 19911 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:16:45.804415 19911 webserver.cc:460] Webserver started at http://127.16.62.66:43469/ using document root <none> and password file <none>
I20260926 23:16:45.805152 19911 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:45.805322 19911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:45.805676 19911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:45.808862 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "f7b151b999204f79b9abdd1445b46158"
format_stamp: "Formatted at 2026-09-26 23:16:45 on dist-test-slave-vcv1"
server_key: "18c8528fa6f264dd1db18a7e083242c8"
server_key_iv: "375cc2293a152cced9aacb5377e6534e"
server_key_version: "encryptionkey@0"
I20260926 23:16:45.809780 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "f7b151b999204f79b9abdd1445b46158"
format_stamp: "Formatted at 2026-09-26 23:16:45 on dist-test-slave-vcv1"
server_key: "18c8528fa6f264dd1db18a7e083242c8"
server_key_iv: "375cc2293a152cced9aacb5377e6534e"
server_key_version: "encryptionkey@0"
I20260926 23:16:45.815256 19911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260926 23:16:45.818655 19926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:45.820042 19911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:16:45.820212 19911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "f7b151b999204f79b9abdd1445b46158"
format_stamp: "Formatted at 2026-09-26 23:16:45 on dist-test-slave-vcv1"
server_key: "18c8528fa6f264dd1db18a7e083242c8"
server_key_iv: "375cc2293a152cced9aacb5377e6534e"
server_key_version: "encryptionkey@0"
I20260926 23:16:45.820436 19911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:45.848949 19911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:45.852547 19911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:45.852793 19911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:45.853601 19911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:45.854801 19911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:45.854872 19911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:45.854956 19911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:45.854998 19911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:45.886916 19911 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:39289
I20260926 23:16:45.886982 20038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:39289 every 8 connection(s)
I20260926 23:16:45.888211 19911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:16:45.895018 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 19911
I20260926 23:16:45.895231 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:16:45.895705 16633 external_mini_cluster.cc:1428] Setting key 32e278a58cd84ef7379ba054221868e2
Sep 26 23:16:45 dist-test-slave-vcv1 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464605, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:45.907497 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:45.890729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:36137 (local address 127.16.62.126:40793)
0926 23:16:45.890927 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:45.890933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:45.892063 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 23:16:45.893806 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:45.893810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:45.893875 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:45.893959 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:45.895426 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:45.896313 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:45.898124 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:45.898538 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:45.898674 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:45.902142 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:45.902147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:45.902150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:45.902187 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:45.904746 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:45.905404 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:45.905411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:45.905414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:45.905536 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:45.905937 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:45.905944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:45.905947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:45.906189 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:45.906328 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:45.906961 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 23:16:45.907183 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:16:45.908496 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:45.891256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:40793 (local address 127.16.62.66:36137)
0926 23:16:45.891918 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:45.891954 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:45.893553 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:45.894154 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:45.894162 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:45.894627 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:45.895251 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:45.895290 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:45.896456 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:45.896462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:45.897976 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:45.897988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:45.898765 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:45.898771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:45.899015 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:45.899702 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:45.899724 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:45.901933 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:45.904874 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:45.904882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:45.904897 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:45.905292 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:45.905647 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:45.905652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:45.905656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:45.905833 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:45.906323 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:45.906331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:45.906626 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 23:16:45.906886 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 23:16:45.907367 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":256,"threads_started":1}
I20260926 23:16:45.909935 20039 heartbeater.cc:347] Connected to a master server at 127.16.62.126:40793
I20260926 23:16:45.910243 20039 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:45.910907 20039 heartbeater.cc:511] Master 127.16.62.126:40793 requested a full tablet report, sending...
I20260926 23:16:45.912338 19716 ts_manager.cc:194] Registered new tserver with Master: f7b151b999204f79b9abdd1445b46158 (127.16.62.66:39289)
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
I20260926 23:16:45.913455 19716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:36137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:16:45.937148 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40793
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:46.244391 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:46.244639 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:46.244727 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:46.251629 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:46.251719 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:46.251822 20044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:16:46.260655 20044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:46.261870 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:46.263095 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:46.276103 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:46.276077 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:46.276152 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:46.276614 20044 server_base.cc:1034] running on GCE node
I20260926 23:16:46.277288 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:46.278110 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:46.279300 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790464606279255 us; error 37 us; skew 500 ppm
I20260926 23:16:46.279601 20044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464606, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:46.282683 20044 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:16:46.284021 20044 webserver.cc:460] Webserver started at http://127.16.62.67:37915/ using document root <none> and password file <none>
I20260926 23:16:46.284703 20044 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:46.284844 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:46.285147 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:46.287911 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "82f3c01f002c40f5bac3e0e612cd55c2"
format_stamp: "Formatted at 2026-09-26 23:16:46 on dist-test-slave-vcv1"
server_key: "b6a84a7722b11f5055ed154be5f3c824"
server_key_iv: "ef64578eaeca6e58a445a5e1397481d4"
server_key_version: "encryptionkey@0"
I20260926 23:16:46.288728 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "82f3c01f002c40f5bac3e0e612cd55c2"
format_stamp: "Formatted at 2026-09-26 23:16:46 on dist-test-slave-vcv1"
server_key: "b6a84a7722b11f5055ed154be5f3c824"
server_key_iv: "ef64578eaeca6e58a445a5e1397481d4"
server_key_version: "encryptionkey@0"
I20260926 23:16:46.293823 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 23:16:46.297106 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:46.298342 20044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:16:46.298501 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "82f3c01f002c40f5bac3e0e612cd55c2"
format_stamp: "Formatted at 2026-09-26 23:16:46 on dist-test-slave-vcv1"
server_key: "b6a84a7722b11f5055ed154be5f3c824"
server_key_iv: "ef64578eaeca6e58a445a5e1397481d4"
server_key_version: "encryptionkey@0"
I20260926 23:16:46.298658 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:46.340559 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:46.344901 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:46.345201 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:46.346120 20044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:46.347414 20044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:46.347483 20044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:46.347625 20044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:46.347680 20044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:46.379972 20044 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:37789
I20260926 23:16:46.380003 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:37789 every 8 connection(s)
I20260926 23:16:46.381492 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:16:46.383172 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 20044
I20260926 23:16:46.383376 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:16:46.383867 16633 external_mini_cluster.cc:1428] Setting key 9c82605d089b357a7fc73f61cfd9e20e
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464606, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:46.404367 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:46.384297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:37277 (local address 127.16.62.126:40793)
0926 23:16:46.384542 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:46.384551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:46.388824 (+ 4273us) server_negotiation.cc:407] Connection header received
0926 23:16:46.391084 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:46.391088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:46.391149 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:46.391231 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:46.392647 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.393348 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.394747 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.395116 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.395202 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:46.398797 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:46.398803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:46.398806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:46.398835 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:46.401592 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.402272 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.402279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.402284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.402399 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.402815 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.402820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.402823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.403079 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:46.403222 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:46.403836 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 23:16:46.404028 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:16:46.405169 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:46.384945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:40793 (local address 127.16.62.67:37277)
0926 23:16:46.388688 (+ 3743us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:46.388736 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:46.390859 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:46.391420 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:46.391427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:46.391884 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:46.392512 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:46.392528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.393476 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.393479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:46.394606 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:46.394616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.395337 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.395342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:46.395512 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:46.396088 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:46.396108 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:46.398563 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:46.401732 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:46.401740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:46.401753 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:46.402154 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:46.402500 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:46.402506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:46.402509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:46.402702 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:46.403199 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:46.403206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:46.403503 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 23:16:46.403740 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 23:16:46.404201 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3448,"thread_start_us":3211,"threads_started":1}
I20260926 23:16:46.406491 20172 heartbeater.cc:347] Connected to a master server at 127.16.62.126:40793
I20260926 23:16:46.406764 20172 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:46.407420 20172 heartbeater.cc:511] Master 127.16.62.126:40793 requested a full tablet report, sending...
I20260926 23:16:46.408727 19716 ts_manager.cc:194] Registered new tserver with Master: 82f3c01f002c40f5bac3e0e612cd55c2 (127.16.62.67:37789)
I20260926 23:16:46.409627 19716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:37277
I20260926 23:16:46.411343 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:16:46.418201 19905 heartbeater.cc:503] Master 127.16.62.126:40793 was elected leader, sending a full tablet report...
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464606, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464606, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:46.456014 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:46.443441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.16.62.126:40793)
0926 23:16:46.443610 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:46.443614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:46.444107 (+ 493us) server_negotiation.cc:407] Connection header received
0926 23:16:46.444322 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:46.444326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:46.444387 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:46.444471 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:46.445722 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.446402 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.447572 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.447981 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.448068 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:46.451708 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:46.451713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:46.451716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:46.451745 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:46.453991 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.454500 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.454503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.454505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.454612 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.454918 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.454921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.454923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.455205 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:46.455363 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:46.455607 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 23:16:46.455796 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 23:16:46.456583 19716 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40838
I20260926 23:16:46.458151 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19777
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464606, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:46.472398 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:46.459141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.16.62.65:33887)
0926 23:16:46.459712 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:46.459719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:46.459744 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:16:46.459792 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:46.459798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:46.459992 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:46.460118 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:46.461257 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.462197 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.463430 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.463945 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.464037 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:46.467418 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:46.467427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:46.467439 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:46.467466 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:46.470246 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.470810 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.470816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.470818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.470928 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.471218 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.471221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.471223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.471510 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:46.471714 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:46.471892 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 23:16:46.472065 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":298,"threads_started":1}
W20260926 23:16:46.473642 19900 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39636
W20260926 23:16:46.474699 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19777: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:16:46.474879 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 19777
I20260926 23:16:46.490116 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19911
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464606, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:16:46.505538 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:46.490941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.16.62.66:39289)
0926 23:16:46.491483 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:46.491490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:46.491516 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:16:46.491577 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:46.491584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:46.491784 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:46.491917 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:46.493123 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.494204 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.495376 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.495973 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.496108 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:46.499939 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:46.499948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:46.499961 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:46.499998 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:46.503015 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.503639 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.503646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.503648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.503776 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.504104 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.504107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.504109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.504571 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:46.504802 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:46.505080 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 23:16:46.505268 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":284,"threads_started":1}
W20260926 23:16:46.506745 20034 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33322
W20260926 23:16:46.507726 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19911: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:16:46.507892 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 19911
I20260926 23:16:46.523559 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 20044
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464606, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:16:46.539029 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:46.524540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.16.62.67:37789)
0926 23:16:46.525097 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:46.525104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:46.525130 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:16:46.525185 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:46.525191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:46.525413 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:46.525529 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:46.527041 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.528038 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.529315 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:46.529865 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:46.529956 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:46.533991 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:46.534000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:46.534014 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:46.534047 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:46.536734 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.537323 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.537329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.537331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.537435 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:46.537735 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:46.537738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:46.537740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:46.538031 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:46.538254 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:46.538471 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 23:16:46.538650 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":265,"threads_started":1}
W20260926 23:16:46.540341 20167 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50938
W20260926 23:16:46.541369 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 20044: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:16:46.541518 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 20044
I20260926 23:16:46.556447 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 19684
I20260926 23:16:46.557597 19746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:46.657027 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 19684
2026-09-26T23:16:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3502 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[20189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[20189](info): set up 2 sockets
Sep 26 23:16:46 dist-test-slave-vcv1 krb5kdc[20189](info): commencing operation
krb5kdc: starting...
W20260926 23:16:47.733556 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:16:47 dist-test-slave-vcv1 krb5kdc[20189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:16:47.921832 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:42879
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:48.230842 20202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:48.231122 20202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:48.231287 20202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:48.238410 20202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:48.238503 20202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:48.238616 20202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:48.238744 20202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:48.238821 20202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:48.248446 20202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:42879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:48.249934 20202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:48.251322 20202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:48.263188 20208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:48.263259 20210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:48.263482 20207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:48.263864 20202 server_base.cc:1034] running on GCE node
I20260926 23:16:48.264672 20202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:48.265996 20202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:48.267246 20202 hybrid_clock.cc:648] HybridClock initialized: now 1790464608267200 us; error 56 us; skew 500 ppm
I20260926 23:16:48.267616 20202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:48 dist-test-slave-vcv1 krb5kdc[20189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464608, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:48.277853 20202 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:16:48.279255 20202 webserver.cc:460] Webserver started at http://127.16.62.126:39055/ using document root <none> and password file <none>
I20260926 23:16:48.279932 20202 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:48.280076 20202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:48.280440 20202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:48.283289 20202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "416557e536bf4803b7659bf28b0a19cd"
format_stamp: "Formatted at 2026-09-26 23:16:48 on dist-test-slave-vcv1"
server_key: "6a2b30bb10a949ebd860085c2623e04c"
server_key_iv: "929c8d7a03449dd29bcbe19d3f8a765b"
server_key_version: "encryptionkey@0"
I20260926 23:16:48.284050 20202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "416557e536bf4803b7659bf28b0a19cd"
format_stamp: "Formatted at 2026-09-26 23:16:48 on dist-test-slave-vcv1"
server_key: "6a2b30bb10a949ebd860085c2623e04c"
server_key_iv: "929c8d7a03449dd29bcbe19d3f8a765b"
server_key_version: "encryptionkey@0"
I20260926 23:16:48.289179 20202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 23:16:48.292488 20217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:48.293797 20202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 23:16:48.293987 20202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "416557e536bf4803b7659bf28b0a19cd"
format_stamp: "Formatted at 2026-09-26 23:16:48 on dist-test-slave-vcv1"
server_key: "6a2b30bb10a949ebd860085c2623e04c"
server_key_iv: "929c8d7a03449dd29bcbe19d3f8a765b"
server_key_version: "encryptionkey@0"
I20260926 23:16:48.294188 20202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:48.317016 20202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:48.321061 20202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:48.321341 20202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:48.339200 20202 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:42879
I20260926 23:16:48.339211 20268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:42879 every 8 connection(s)
I20260926 23:16:48.340596 20202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:48.344214 20269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:48.348557 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 20202
I20260926 23:16:48.348765 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:48.349529 16633 external_mini_cluster.cc:1428] Setting key 40011a913a8363c1f24a22760c09ca66
I20260926 23:16:48.355472 20269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd: Bootstrap starting.
I20260926 23:16:48.358945 20269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:48.360041 20269 log.cc:824] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:48.363270 20269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd: No bootstrap required, opened a new log
Sep 26 23:16:48 dist-test-slave-vcv1 krb5kdc[20189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:48.373283 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:48.351473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.16.62.126:42879)
0926 23:16:48.352294 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:48.352306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:48.352343 (+ 37us) server_negotiation.cc:407] Connection header received
0926 23:16:48.354022 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:48.354043 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:48.354403 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:48.354700 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:48.356490 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:48.357766 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:48.359466 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:48.360111 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:48.360309 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:48.365083 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:48.365093 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:48.365116 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:48.365160 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:48.369117 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:48.369713 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:48.369721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:48.369729 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:48.369939 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:48.370272 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:48.370277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:48.370281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:48.370958 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:48.371243 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:48.371540 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 23:16:48.371983 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":378,"threads_started":1}
I20260926 23:16:48.373900 20269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416557e536bf4803b7659bf28b0a19cd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42879 } }
I20260926 23:16:48.374222 20269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:48.374332 20269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416557e536bf4803b7659bf28b0a19cd, State: Initialized, Role: FOLLOWER
I20260926 23:16:48.374915 20269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416557e536bf4803b7659bf28b0a19cd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42879 } }
I20260926 23:16:48.375139 20269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:48.375298 20269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:48.375470 20269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:48.377641 20269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416557e536bf4803b7659bf28b0a19cd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42879 } }
I20260926 23:16:48.378099 20269 leader_election.cc:302] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 416557e536bf4803b7659bf28b0a19cd; no voters:
I20260926 23:16:48.378675 20269 leader_election.cc:288] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:48.378803 20274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:48.379478 20274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [term 1 LEADER]: Becoming Leader. State: Replica: 416557e536bf4803b7659bf28b0a19cd, State: Running, Role: LEADER
I20260926 23:16:48.379945 20274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416557e536bf4803b7659bf28b0a19cd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42879 } }
I20260926 23:16:48.380582 20269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:48.382772 20276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416557e536bf4803b7659bf28b0a19cd. Latest consensus state: current_term: 1 leader_uuid: "416557e536bf4803b7659bf28b0a19cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416557e536bf4803b7659bf28b0a19cd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42879 } } }
I20260926 23:16:48.382874 20275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "416557e536bf4803b7659bf28b0a19cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416557e536bf4803b7659bf28b0a19cd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42879 } } }
I20260926 23:16:48.383239 20275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:48.383128 20276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:48.384389 20282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:48.389279 20282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:48.397917 20282 catalog_manager.cc:1347] Generated new cluster ID: 0f970620d99346259837630eefa4887c
I20260926 23:16:48.398012 20282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:48.421758 20282 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:48.422988 20282 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:48.438328 20282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 416557e536bf4803b7659bf28b0a19cd: Generated new TSK 0
I20260926 23:16:48.439188 20282 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:16:48.495851 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42879
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:48.825620 20295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:48.825912 20295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:48.826129 20295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:48.834144 20295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:48.834251 20295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:48.834400 20295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:16:48.843869 20295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:48.845130 20295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:48.846426 20295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:48.859423 20301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:48.859445 20303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:48.859645 20300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:48.860409 20295 server_base.cc:1034] running on GCE node
I20260926 23:16:48.860991 20295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:48.861822 20295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:48.863006 20295 hybrid_clock.cc:648] HybridClock initialized: now 1790464608862967 us; error 39 us; skew 500 ppm
I20260926 23:16:48.863303 20295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:48 dist-test-slave-vcv1 krb5kdc[20189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464608, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:48.866178 20295 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:16:48.867453 20295 webserver.cc:460] Webserver started at http://127.16.62.65:38687/ using document root <none> and password file <none>
I20260926 23:16:48.868058 20295 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:48.868209 20295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:48.868496 20295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:48.871272 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "749c8fe98c3b424d887bbf349e15e310"
format_stamp: "Formatted at 2026-09-26 23:16:48 on dist-test-slave-vcv1"
server_key: "15b7037b97787f64fadc1dff05d82447"
server_key_iv: "bd8e1907dbd39ef5b5fc5dcf7996507e"
server_key_version: "encryptionkey@0"
I20260926 23:16:48.872148 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "749c8fe98c3b424d887bbf349e15e310"
format_stamp: "Formatted at 2026-09-26 23:16:48 on dist-test-slave-vcv1"
server_key: "15b7037b97787f64fadc1dff05d82447"
server_key_iv: "bd8e1907dbd39ef5b5fc5dcf7996507e"
server_key_version: "encryptionkey@0"
I20260926 23:16:48.877070 20295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:16:48.880281 20310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:48.881680 20295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:16:48.881834 20295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "749c8fe98c3b424d887bbf349e15e310"
format_stamp: "Formatted at 2026-09-26 23:16:48 on dist-test-slave-vcv1"
server_key: "15b7037b97787f64fadc1dff05d82447"
server_key_iv: "bd8e1907dbd39ef5b5fc5dcf7996507e"
server_key_version: "encryptionkey@0"
I20260926 23:16:48.882045 20295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:48.899688 20295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:48.903498 20295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:48.903728 20295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:48.904659 20295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:48.905894 20295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:48.905966 20295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:48.906049 20295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:48.906092 20295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:48.938036 20295 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:42191
I20260926 23:16:48.938150 20422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:42191 every 8 connection(s)
I20260926 23:16:48.939321 20295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:16:48.942400 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 20295
I20260926 23:16:48.942601 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:16:48.943432 16633 external_mini_cluster.cc:1428] Setting key 3f9d2951bd52554ed0f637d52ff20e6d
Sep 26 23:16:48 dist-test-slave-vcv1 krb5kdc[20189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464608, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:48.959002 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:48.941760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:40801 (local address 127.16.62.126:42879)
0926 23:16:48.942154 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:48.942158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:48.943208 (+ 1050us) server_negotiation.cc:407] Connection header received
0926 23:16:48.945149 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:48.945153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:48.945251 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:48.945358 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:48.946731 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:48.947845 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:48.949291 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:48.949956 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:48.950085 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:48.953681 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:48.953685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:48.953689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:48.953717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:48.956191 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:48.956855 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:48.956859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:48.956863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:48.956977 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:48.957386 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:48.957390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:48.957394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:48.957627 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:48.957778 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:48.958425 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 23:16:48.958627 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":214,"threads_started":1}
I20260926 23:16:48.959905 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:48.942310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:42879 (local address 127.16.62.65:40801)
0926 23:16:48.943068 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:48.943103 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:48.944885 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:48.945542 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:48.945549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:48.946009 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:48.946534 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:48.946550 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:48.947998 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:48.948001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:48.949135 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:48.949163 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:48.950146 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:48.950150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:48.950330 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:48.951068 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:48.951104 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:48.953477 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:48.956326 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:48.956334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:48.956348 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:48.956733 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:48.957081 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:48.957087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:48.957091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:48.957276 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:48.957755 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:48.957762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:48.958057 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 23:16:48.958327 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 23:16:48.958793 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":338,"threads_started":1}
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
I20260926 23:16:48.961431 20423 heartbeater.cc:347] Connected to a master server at 127.16.62.126:42879
I20260926 23:16:48.961719 20423 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:48.962450 20423 heartbeater.cc:511] Master 127.16.62.126:42879 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
I20260926 23:16:48.964681 20234 ts_manager.cc:194] Registered new tserver with Master: 749c8fe98c3b424d887bbf349e15e310 (127.16.62.65:42191)
I20260926 23:16:48.966885 20234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:40801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:16:48.987419 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42879
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:49.306890 20429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:49.307188 20429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:49.307289 20429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:49.314778 20429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:49.314898 20429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:49.315024 20429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:16:49.324710 20429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:49.325932 20429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:49.327251 20429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:49.340926 20437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:49.340970 20434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:49.343765 20435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:49.345534 20429 server_base.cc:1034] running on GCE node
I20260926 23:16:49.346189 20429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:49.347159 20429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:49.348376 20429 hybrid_clock.cc:648] HybridClock initialized: now 1790464609348323 us; error 46 us; skew 500 ppm
I20260926 23:16:49.348686 20429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:49 dist-test-slave-vcv1 krb5kdc[20189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:49.351672 20429 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:16:49.353269 20429 webserver.cc:460] Webserver started at http://127.16.62.66:45163/ using document root <none> and password file <none>
I20260926 23:16:49.353955 20429 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:49.354082 20429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:49.354389 20429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:49.357208 20429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "3e698026b926400bbadc54eabeee8815"
format_stamp: "Formatted at 2026-09-26 23:16:49 on dist-test-slave-vcv1"
server_key: "323e9119cdebf19e0ac8569c75ec7216"
server_key_iv: "7ae61f47962bde54a2871709a7c470bb"
server_key_version: "encryptionkey@0"
I20260926 23:16:49.357937 20429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "3e698026b926400bbadc54eabeee8815"
format_stamp: "Formatted at 2026-09-26 23:16:49 on dist-test-slave-vcv1"
server_key: "323e9119cdebf19e0ac8569c75ec7216"
server_key_iv: "7ae61f47962bde54a2871709a7c470bb"
server_key_version: "encryptionkey@0"
I20260926 23:16:49.362831 20429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:16:49.366094 20444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:49.367265 20429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:16:49.367424 20429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "3e698026b926400bbadc54eabeee8815"
format_stamp: "Formatted at 2026-09-26 23:16:49 on dist-test-slave-vcv1"
server_key: "323e9119cdebf19e0ac8569c75ec7216"
server_key_iv: "7ae61f47962bde54a2871709a7c470bb"
server_key_version: "encryptionkey@0"
I20260926 23:16:49.367607 20429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:49.387646 20429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:49.391618 20429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:49.391901 20429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:49.392828 20429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:49.394002 20429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:49.394068 20429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:49.394227 20429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:49.394284 20429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:49.425972 20429 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:40699
I20260926 23:16:49.426023 20556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:40699 every 8 connection(s)
I20260926 23:16:49.427227 20429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:16:49.435011 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 20429
I20260926 23:16:49.435191 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:16:49.435603 16633 external_mini_cluster.cc:1428] Setting key 1814bb33e7c1dbb420e27cb65fc6583c
Sep 26 23:16:49 dist-test-slave-vcv1 krb5kdc[20189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:49.446488 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:49.429698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:40663 (local address 127.16.62.126:42879)
0926 23:16:49.429874 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:49.429879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:49.430856 (+ 977us) server_negotiation.cc:407] Connection header received
0926 23:16:49.432635 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:49.432640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:49.432733 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:49.432843 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:49.434228 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.434951 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.436812 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.437286 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.437418 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:49.440770 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:49.440775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:49.440778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:49.440806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:49.443406 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:49.444306 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:49.444310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:49.444313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:49.444432 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:49.444908 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:49.444911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:49.444913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:49.445189 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:49.445311 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:49.446100 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 23:16:49.446263 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:16:49.447925 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:49.430031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:42879 (local address 127.16.62.66:40663)
0926 23:16:49.430682 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:49.430722 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:49.432375 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:49.433061 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:49.433069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:49.433533 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:49.434061 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:49.434077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.435096 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.435101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:49.436676 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:49.436686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.437497 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.437504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:49.437729 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:49.438417 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:49.438442 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:49.440546 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:49.443547 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:49.443558 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:49.443575 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:49.444186 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:49.444534 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:49.444541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:49.444547 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:49.444802 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:49.445326 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:49.445336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:49.445751 (+ 415us) client_negotiation.cc:769] Sending connection context
0926 23:16:49.446033 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 23:16:49.446539 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":258,"threads_started":1}
I20260926 23:16:49.449324 20557 heartbeater.cc:347] Connected to a master server at 127.16.62.126:42879
I20260926 23:16:49.449607 20557 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:49.450304 20557 heartbeater.cc:511] Master 127.16.62.126:42879 requested a full tablet report, sending...
I20260926 23:16:49.451676 20234 ts_manager.cc:194] Registered new tserver with Master: 3e698026b926400bbadc54eabeee8815 (127.16.62.66:40699)
I20260926 23:16:49.452526 20234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:40663
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:16:49.480217 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42879
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:49.802922 20562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:49.803176 20562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:49.803279 20562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:49.810547 20562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:49.810648 20562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:49.810756 20562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:16:49.820282 20562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:49.821527 20562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:49.822777 20562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:49.835171 20567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:49.835315 20568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:49.835196 20570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:49.835968 20562 server_base.cc:1034] running on GCE node
I20260926 23:16:49.836613 20562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:49.837503 20562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:49.838724 20562 hybrid_clock.cc:648] HybridClock initialized: now 1790464609838699 us; error 38 us; skew 500 ppm
I20260926 23:16:49.839025 20562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:49 dist-test-slave-vcv1 krb5kdc[20189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:49.841951 20562 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:16:49.843264 20562 webserver.cc:460] Webserver started at http://127.16.62.67:45763/ using document root <none> and password file <none>
I20260926 23:16:49.843907 20562 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:49.844044 20562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:49.844394 20562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:49.846956 20562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "d8a77377d90d4ff7b8b602b67e9e3dd8"
format_stamp: "Formatted at 2026-09-26 23:16:49 on dist-test-slave-vcv1"
server_key: "e0c640e7f7ea02e4d41b40b41eaf3804"
server_key_iv: "183d18c50fad3e688dfae68ff9269f34"
server_key_version: "encryptionkey@0"
I20260926 23:16:49.847668 20562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "d8a77377d90d4ff7b8b602b67e9e3dd8"
format_stamp: "Formatted at 2026-09-26 23:16:49 on dist-test-slave-vcv1"
server_key: "e0c640e7f7ea02e4d41b40b41eaf3804"
server_key_iv: "183d18c50fad3e688dfae68ff9269f34"
server_key_version: "encryptionkey@0"
I20260926 23:16:49.852349 20562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 23:16:49.855531 20577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:49.856942 20562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:49.857112 20562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "d8a77377d90d4ff7b8b602b67e9e3dd8"
format_stamp: "Formatted at 2026-09-26 23:16:49 on dist-test-slave-vcv1"
server_key: "e0c640e7f7ea02e4d41b40b41eaf3804"
server_key_iv: "183d18c50fad3e688dfae68ff9269f34"
server_key_version: "encryptionkey@0"
I20260926 23:16:49.857299 20562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:49.883368 20562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:49.887156 20562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:49.887435 20562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:49.888352 20562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:49.889588 20562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:49.889695 20562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:49.889837 20562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:49.889910 20562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:49.921859 20562 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:43795
I20260926 23:16:49.921905 20689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:43795 every 8 connection(s)
I20260926 23:16:49.923069 20562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:16:49.927372 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 20562
I20260926 23:16:49.927553 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:16:49.928100 16633 external_mini_cluster.cc:1428] Setting key caec6acdddc028cefe316a9e3485122e
Sep 26 23:16:49 dist-test-slave-vcv1 krb5kdc[20189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:49.942766 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:49.925525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:44295 (local address 127.16.62.126:42879)
0926 23:16:49.925691 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:49.925695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:49.926805 (+ 1110us) server_negotiation.cc:407] Connection header received
0926 23:16:49.929293 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:49.929298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:49.929385 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:49.929489 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:49.931158 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.931883 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.933403 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.933790 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.933950 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:49.937274 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:49.937279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:49.937282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:49.937309 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:49.939740 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:49.940509 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:49.940519 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:49.940521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:49.940636 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:49.941040 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:49.941045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:49.941048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:49.941411 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:49.941554 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:49.942268 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 23:16:49.942473 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:16:49.943766 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:49.925998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:42879 (local address 127.16.62.67:44295)
0926 23:16:49.926635 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:49.926676 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:49.929045 (+ 2369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:49.929993 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:49.930001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:49.930460 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:49.930986 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:49.931003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.932022 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.932027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:49.933262 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:49.933278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.934029 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.934038 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:49.934266 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:49.934954 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:49.934973 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:49.937060 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:49.939868 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:49.939877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:49.939890 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:49.940340 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:49.940749 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:49.940754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:49.940757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:49.940920 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:49.941540 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:49.941547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:49.941880 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 23:16:49.942154 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 23:16:49.942623 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":261,"threads_started":1}
I20260926 23:16:49.944901 20690 heartbeater.cc:347] Connected to a master server at 127.16.62.126:42879
I20260926 23:16:49.945163 20690 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:49.945780 20690 heartbeater.cc:511] Master 127.16.62.126:42879 requested a full tablet report, sending...
I20260926 23:16:49.947016 20234 ts_manager.cc:194] Registered new tserver with Master: d8a77377d90d4ff7b8b602b67e9e3dd8 (127.16.62.67:43795)
I20260926 23:16:49.947772 20234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:44295
I20260926 23:16:49.955113 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:16:49.969805 20423 heartbeater.cc:503] Master 127.16.62.126:42879 was elected leader, sending a full tablet report...
Sep 26 23:16:49 dist-test-slave-vcv1 krb5kdc[20189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:16:49 dist-test-slave-vcv1 krb5kdc[20189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:50.001978 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:49.987168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.16.62.65:42191)
0926 23:16:49.987788 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:49.987794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:49.987817 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:16:49.987870 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:49.987876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:49.988086 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:49.988290 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:49.989646 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.990648 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.991875 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:49.992495 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:49.992634 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:49.996421 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:49.996431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:49.996444 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:49.996474 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:49.999241 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:49.999861 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:49.999873 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:49.999876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:49.999988 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:50.000440 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:50.000448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:50.000452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:50.000785 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:50.001089 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:50.001388 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 23:16:50.001637 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":272,"threads_started":1}
Sep 26 23:16:50 dist-test-slave-vcv1 krb5kdc[20189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:50.019158 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:50.005971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37974 (local address 127.16.62.126:42879)
0926 23:16:50.006183 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:50.006192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:50.006435 (+ 243us) server_negotiation.cc:407] Connection header received
0926 23:16:50.006610 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:50.006614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:50.006680 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:50.006782 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:50.008009 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:50.008797 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:50.010218 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:50.010620 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:50.010746 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:50.014410 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:50.014415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:50.014418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:50.014446 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:50.016793 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:50.017398 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:50.017405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:50.017412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:50.017524 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:50.017869 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:50.017874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:50.017878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:50.018155 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:50.018401 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:50.018671 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 23:16:50.018872 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 23:16:50.019649 20234 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37974
I20260926 23:16:50.020581 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 20295
I20260926 23:16:50.032521 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:50.021408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.16.62.65:42191)
0926 23:16:50.021609 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:50.021615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:50.021839 (+ 224us) server_negotiation.cc:407] Connection header received
0926 23:16:50.022027 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:50.022031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:50.022086 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:50.022182 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:50.023320 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:50.023975 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:50.025172 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:50.025573 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:50.025698 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:50.027767 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:50.027772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:50.027775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:50.027803 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:50.030163 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:50.030742 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:50.030747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:50.030756 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:50.030861 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:50.031241 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:50.031246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:50.031249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:50.031472 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:50.031705 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:50.031989 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 23:16:50.032216 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:16:50.033063 20418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:50.114336 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 20295
I20260926 23:16:50.130652 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 20429
Sep 26 23:16:50 dist-test-slave-vcv1 krb5kdc[20189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:16:50.145869 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:50.131638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.16.62.66:40699)
0926 23:16:50.132237 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:50.132244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:50.132270 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:16:50.132329 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:50.132335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:50.132538 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:50.132678 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:50.133735 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:50.134718 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:50.135885 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:50.136722 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:50.136817 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:50.140626 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:50.140635 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:50.140649 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:50.140709 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:50.143472 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:50.144008 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:50.144014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:50.144016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:50.144126 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:50.144568 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:50.144572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:50.144573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:50.144884 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:50.145088 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:50.145303 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 23:16:50.145481 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":280,"threads_started":1}
I20260926 23:16:50.147167 20552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:50.226984 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 20429
I20260926 23:16:50.242813 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 20562
Sep 26 23:16:50 dist-test-slave-vcv1 krb5kdc[20189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:16:50.258289 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:50.243835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.16.62.67:43795)
0926 23:16:50.244571 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:50.244578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:50.244606 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:16:50.244671 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:50.244678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:50.244895 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:50.245014 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:50.246175 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:50.247193 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:50.248508 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:50.249040 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:50.249139 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:50.253097 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:50.253107 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:50.253138 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:50.253171 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:50.255909 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:50.256533 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:50.256540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:50.256542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:50.256668 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:50.256959 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:50.256961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:50.256963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:50.257294 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:50.257516 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:50.257719 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 23:16:50.257899 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":446,"threads_started":1}
I20260926 23:16:50.259557 20685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:50.340515 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 20562
I20260926 23:16:50.355438 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 20202
I20260926 23:16:50.356694 20264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:50.456230 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 20202
2026-09-26T23:16:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3798 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:16:50 dist-test-slave-vcv1 krb5kdc[20709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:16:50 dist-test-slave-vcv1 krb5kdc[20709](info): set up 2 sockets
Sep 26 23:16:50 dist-test-slave-vcv1 krb5kdc[20709](info): commencing operation
krb5kdc: starting...
W20260926 23:16:51.545579 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:16:51 dist-test-slave-vcv1 krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:16:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:16:51.736056 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41675
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:43475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:52.048393 20722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:52.048671 20722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:52.048844 20722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:52.055898 20722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:52.055987 20722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:52.056077 20722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:52.056195 20722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:52.056277 20722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:52.065810 20722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:41675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:52.067034 20722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:52.068297 20722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:52.079584 20728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:52.079603 20727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:52.080197 20722 server_base.cc:1034] running on GCE node
W20260926 23:16:52.080276 20730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:52.081102 20722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:52.082442 20722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:52.083679 20722 hybrid_clock.cc:648] HybridClock initialized: now 1790464612083655 us; error 55 us; skew 500 ppm
I20260926 23:16:52.084039 20722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:52 dist-test-slave-vcv1 krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464612, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:52.087109 20722 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:16:52.088243 20722 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:52.088330 20722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:52.088595 20722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:52.091353 20722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "b1eba38fffa74a1195763118c3d7aa05"
format_stamp: "Formatted at 2026-09-26 23:16:52 on dist-test-slave-vcv1"
server_key: "cc656eeb5ffd04af49306a98f8094aaf"
server_key_iv: "9e8a9e9a1742e9b9b6d1fb1803d0c9ba"
server_key_version: "encryptionkey@0"
I20260926 23:16:52.092073 20722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "b1eba38fffa74a1195763118c3d7aa05"
format_stamp: "Formatted at 2026-09-26 23:16:52 on dist-test-slave-vcv1"
server_key: "cc656eeb5ffd04af49306a98f8094aaf"
server_key_iv: "9e8a9e9a1742e9b9b6d1fb1803d0c9ba"
server_key_version: "encryptionkey@0"
I20260926 23:16:52.096782 20722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:16:52.100034 20736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:52.101284 20722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:16:52.101460 20722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "b1eba38fffa74a1195763118c3d7aa05"
format_stamp: "Formatted at 2026-09-26 23:16:52 on dist-test-slave-vcv1"
server_key: "cc656eeb5ffd04af49306a98f8094aaf"
server_key_iv: "9e8a9e9a1742e9b9b6d1fb1803d0c9ba"
server_key_version: "encryptionkey@0"
I20260926 23:16:52.101673 20722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:52.133102 20722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:52.137387 20722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:52.137662 20722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:52.155447 20722 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:41675
I20260926 23:16:52.155457 20787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:41675 every 8 connection(s)
I20260926 23:16:52.156335 20722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:52.159756 20788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:52.166599 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 20722
I20260926 23:16:52.166787 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:52.167547 16633 external_mini_cluster.cc:1428] Setting key e64f44c175d72e85631a40b2d2236085
I20260926 23:16:52.170450 20788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05: Bootstrap starting.
I20260926 23:16:52.173743 20788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:52.174885 20788 log.cc:824] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:52.177687 20788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05: No bootstrap required, opened a new log
Sep 26 23:16:52 dist-test-slave-vcv1 krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:52.186494 20788 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1eba38fffa74a1195763118c3d7aa05" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41675 } }
I20260926 23:16:52.186960 20788 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:52.187065 20788 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1eba38fffa74a1195763118c3d7aa05, State: Initialized, Role: FOLLOWER
I20260926 23:16:52.187711 20788 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1eba38fffa74a1195763118c3d7aa05" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41675 } }
I20260926 23:16:52.187987 20788 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:52.188155 20788 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:52.188351 20788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:52.189643 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:52.169482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.16.62.126:41675)
0926 23:16:52.170175 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:52.170185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:52.170232 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:16:52.171926 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:52.171946 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:52.172373 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:52.172658 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:52.174446 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:52.175794 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:52.177119 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:52.178123 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:52.178317 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:52.182437 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:52.182447 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:52.182462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:52.182505 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:52.185492 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:52.186257 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:52.186263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:52.186267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:52.186412 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:52.186792 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:52.186794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:52.186796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:52.187470 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:52.187728 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:52.188278 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 23:16:52.188661 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":323,"threads_started":1}
I20260926 23:16:52.190761 20788 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1eba38fffa74a1195763118c3d7aa05" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41675 } }
I20260926 23:16:52.191331 20788 leader_election.cc:302] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1eba38fffa74a1195763118c3d7aa05; no voters:
I20260926 23:16:52.191913 20788 leader_election.cc:288] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:52.192026 20793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:52.193667 20788 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:52.194547 20793 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [term 1 LEADER]: Becoming Leader. State: Replica: b1eba38fffa74a1195763118c3d7aa05, State: Running, Role: LEADER
I20260926 23:16:52.195070 20793 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1eba38fffa74a1195763118c3d7aa05" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41675 } }
I20260926 23:16:52.198388 20794 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1eba38fffa74a1195763118c3d7aa05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1eba38fffa74a1195763118c3d7aa05" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41675 } } }
I20260926 23:16:52.198238 20795 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1eba38fffa74a1195763118c3d7aa05. Latest consensus state: current_term: 1 leader_uuid: "b1eba38fffa74a1195763118c3d7aa05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1eba38fffa74a1195763118c3d7aa05" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41675 } } }
I20260926 23:16:52.199316 20794 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:52.199323 20795 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:52.200829 20805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:52.204978 20805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:52.212852 20805 catalog_manager.cc:1347] Generated new cluster ID: 69caa6cef6004a7e840a65b8893f7845
I20260926 23:16:52.212949 20805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:52.225668 20805 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:52.227391 20805 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:52.242199 20805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05: Generated new TSK 0
I20260926 23:16:52.243165 20805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:16:52.287160 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41675
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:43475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:52.611753 20814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:52.612012 20814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:52.612118 20814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:52.619401 20814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:52.619516 20814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:52.619637 20814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:16:52.629124 20814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:52.630414 20814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:52.631631 20814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:52.644204 20822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:52.644274 20820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:52.644335 20819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:52.644853 20814 server_base.cc:1034] running on GCE node
I20260926 23:16:52.645433 20814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:52.646275 20814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:52.647482 20814 hybrid_clock.cc:648] HybridClock initialized: now 1790464612647443 us; error 53 us; skew 500 ppm
I20260926 23:16:52.647791 20814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:52 dist-test-slave-vcv1 krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464612, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:52.650774 20814 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:16:52.651818 20814 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:52.651928 20814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:52.652297 20814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:52.655064 20814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "9ea92c4329b646dbaf55add74452cbaf"
format_stamp: "Formatted at 2026-09-26 23:16:52 on dist-test-slave-vcv1"
server_key: "c21580efeb7c5760a8e04b41e5ef0aff"
server_key_iv: "c966c40067bed237b7a6d9295acd3966"
server_key_version: "encryptionkey@0"
I20260926 23:16:52.655875 20814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "9ea92c4329b646dbaf55add74452cbaf"
format_stamp: "Formatted at 2026-09-26 23:16:52 on dist-test-slave-vcv1"
server_key: "c21580efeb7c5760a8e04b41e5ef0aff"
server_key_iv: "c966c40067bed237b7a6d9295acd3966"
server_key_version: "encryptionkey@0"
I20260926 23:16:52.660888 20814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 23:16:52.664352 20828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:52.665635 20814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:16:52.665795 20814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "9ea92c4329b646dbaf55add74452cbaf"
format_stamp: "Formatted at 2026-09-26 23:16:52 on dist-test-slave-vcv1"
server_key: "c21580efeb7c5760a8e04b41e5ef0aff"
server_key_iv: "c966c40067bed237b7a6d9295acd3966"
server_key_version: "encryptionkey@0"
I20260926 23:16:52.666010 20814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:52.683835 20814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:52.687492 20814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:52.687758 20814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:52.688696 20814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:52.689903 20814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:52.689983 20814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:52.690138 20814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:52.690232 20814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:52.723428 20814 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:35221
I20260926 23:16:52.723482 20940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:35221 every 8 connection(s)
I20260926 23:16:52.724332 20814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:16:52.725194 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 20814
I20260926 23:16:52.725437 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:16:52.726094 16633 external_mini_cluster.cc:1428] Setting key e83faac5c1567d4a82ca616bcfc520d5
Sep 26 23:16:52 dist-test-slave-vcv1 krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464612, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:52.744231 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:52.727097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:41983 (local address 127.16.62.126:41675)
0926 23:16:52.727481 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:52.727486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:52.728700 (+ 1214us) server_negotiation.cc:407] Connection header received
0926 23:16:52.730415 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:52.730420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:52.730504 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:52.730611 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:52.732034 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:52.732826 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:52.734333 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:52.734737 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:52.734867 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:52.738192 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:52.738197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:52.738200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:52.738231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:52.740861 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:52.741672 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:52.741679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:52.741682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:52.741854 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:52.742397 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:52.742407 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:52.742411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:52.742724 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:52.742837 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:52.743771 (+ 934us) server_negotiation.cc:299] Negotiation successful
0926 23:16:52.743926 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":188,"threads_started":1}
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
I20260926 23:16:52.745394 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:52.727710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41675 (local address 127.16.62.65:41983)
0926 23:16:52.728559 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:52.728607 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:52.730167 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:52.730846 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:52.730854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:52.731357 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:52.731877 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:52.731893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:52.732968 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:52.732973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:52.734180 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:52.734191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:52.734952 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:52.734958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:52.735174 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:52.735905 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:52.735925 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:52.737991 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:52.740997 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:52.741005 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:52.741019 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:52.741557 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:52.742009 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:52.742014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:52.742018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:52.742257 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:52.742888 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:52.742897 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:52.743358 (+ 461us) client_negotiation.cc:769] Sending connection context
0926 23:16:52.743709 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 23:16:52.744278 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":344,"threads_started":1}
I20260926 23:16:52.746977 20941 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
I20260926 23:16:52.747311 20941 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:52.747962 20941 heartbeater.cc:511] Master 127.16.62.126:41675 requested a full tablet report, sending...
I20260926 23:16:52.749924 20753 ts_manager.cc:194] Registered new tserver with Master: 9ea92c4329b646dbaf55add74452cbaf (127.16.62.65:35221)
I20260926 23:16:52.751277 20753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:41983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:16:52.770843 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41675
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:43475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:53.085386 20947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:53.085789 20947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:53.085965 20947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:53.093497 20947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:53.093616 20947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:53.093765 20947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:16:53.103605 20947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:53.104887 20947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:53.106289 20947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:53.118806 20952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:53.118867 20955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:53.118834 20953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:53.119841 20947 server_base.cc:1034] running on GCE node
I20260926 23:16:53.120484 20947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:53.121373 20947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:53.122596 20947 hybrid_clock.cc:648] HybridClock initialized: now 1790464613122551 us; error 53 us; skew 500 ppm
I20260926 23:16:53.122898 20947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:53 dist-test-slave-vcv1 krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:53.126030 20947 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:16:53.127051 20947 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:53.127135 20947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:53.127362 20947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:53.130157 20947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36"
format_stamp: "Formatted at 2026-09-26 23:16:53 on dist-test-slave-vcv1"
server_key: "3d246909f2daffb7e8eb70bc94a2c1f7"
server_key_iv: "1644d09d86b974f0d172cfd2a7f556fc"
server_key_version: "encryptionkey@0"
I20260926 23:16:53.130911 20947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36"
format_stamp: "Formatted at 2026-09-26 23:16:53 on dist-test-slave-vcv1"
server_key: "3d246909f2daffb7e8eb70bc94a2c1f7"
server_key_iv: "1644d09d86b974f0d172cfd2a7f556fc"
server_key_version: "encryptionkey@0"
I20260926 23:16:53.135521 20947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 23:16:53.138674 20961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:53.139899 20947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 23:16:53.140021 20947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36"
format_stamp: "Formatted at 2026-09-26 23:16:53 on dist-test-slave-vcv1"
server_key: "3d246909f2daffb7e8eb70bc94a2c1f7"
server_key_iv: "1644d09d86b974f0d172cfd2a7f556fc"
server_key_version: "encryptionkey@0"
I20260926 23:16:53.140252 20947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:53.159425 20947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:53.163602 20947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:53.163903 20947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:53.164875 20947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:53.166142 20947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:53.166258 20947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:53.166391 20947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:53.166488 20947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:53.198952 20947 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:34803
I20260926 23:16:53.199036 21073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:34803 every 8 connection(s)
I20260926 23:16:53.199788 20947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:16:53.207152 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 20947
I20260926 23:16:53.207360 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:16:53.207862 16633 external_mini_cluster.cc:1428] Setting key 170e4323d8f0d59dc2c15a96be88ebdd
Sep 26 23:16:53 dist-test-slave-vcv1 krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:53.219781 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.202086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:55813 (local address 127.16.62.126:41675)
0926 23:16:53.202263 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.202268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.203437 (+ 1169us) server_negotiation.cc:407] Connection header received
0926 23:16:53.205220 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.205224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.205349 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.205430 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:53.206811 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.207671 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.209429 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.209939 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.210067 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.213395 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:53.213404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:53.213412 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:53.213487 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:53.216684 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:53.217617 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:53.217621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:53.217623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:53.217762 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:53.218225 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:53.218229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:53.218231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:53.218469 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:53.218570 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.219383 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.219544 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:16:53.221289 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.202603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41675 (local address 127.16.62.66:55813)
0926 23:16:53.203276 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:53.203319 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:53.204965 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:53.205642 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:53.205651 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:53.206124 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:53.206658 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.206673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.207821 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.207829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.209282 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.209294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.210168 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.210174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.210358 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.211048 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:53.211070 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:53.213186 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:53.216816 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:53.216827 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:53.216846 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:53.217493 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:53.217865 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:53.217873 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:53.217879 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:53.218100 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:53.218595 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:53.218605 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:53.219031 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 23:16:53.219308 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 23:16:53.219861 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":254,"threads_started":1}
I20260926 23:16:53.222733 21074 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41675
I20260926 23:16:53.223028 21074 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:53.223704 21074 heartbeater.cc:511] Master 127.16.62.126:41675 requested a full tablet report, sending...
I20260926 23:16:53.225143 20753 ts_manager.cc:194] Registered new tserver with Master: 12a4d5ce1d2e4c13bef78cf9e19dbe36 (127.16.62.66:34803)
I20260926 23:16:53.225917 20753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:55813
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:16:53.253414 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41675
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:43475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:16:53.565228 21079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:53.565551 21079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:53.565718 21079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:53.573207 21079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:53.573391 21079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:53.573570 21079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:16:53.583112 21079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:53.584407 21079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:53.585655 21079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:53.598392 21084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:53.598526 21085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:53.598506 21087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:53.604557 21079 server_base.cc:1034] running on GCE node
I20260926 23:16:53.605170 21079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:53.606172 21079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:53.607398 21079 hybrid_clock.cc:648] HybridClock initialized: now 1790464613607351 us; error 60 us; skew 500 ppm
I20260926 23:16:53.607755 21079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:16:53 dist-test-slave-vcv1 krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:16:53.618481 21079 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:16:53.619685 21079 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:53.619781 21079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:53.620038 21079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:53.622881 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "a240b945f67c48aeba7a95e69f246acc"
format_stamp: "Formatted at 2026-09-26 23:16:53 on dist-test-slave-vcv1"
server_key: "1d2d7050134e06d99394fa646f5bb45d"
server_key_iv: "8935d3407fa21e548ad33c5a497d8e17"
server_key_version: "encryptionkey@0"
I20260926 23:16:53.623594 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "a240b945f67c48aeba7a95e69f246acc"
format_stamp: "Formatted at 2026-09-26 23:16:53 on dist-test-slave-vcv1"
server_key: "1d2d7050134e06d99394fa646f5bb45d"
server_key_iv: "8935d3407fa21e548ad33c5a497d8e17"
server_key_version: "encryptionkey@0"
I20260926 23:16:53.628496 21079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 23:16:53.631623 21093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:53.633035 21079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:53.633214 21079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "a240b945f67c48aeba7a95e69f246acc"
format_stamp: "Formatted at 2026-09-26 23:16:53 on dist-test-slave-vcv1"
server_key: "1d2d7050134e06d99394fa646f5bb45d"
server_key_iv: "8935d3407fa21e548ad33c5a497d8e17"
server_key_version: "encryptionkey@0"
I20260926 23:16:53.633430 21079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:53.652000 21079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:53.655700 21079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:53.655987 21079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:53.656972 21079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:53.658257 21079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:53.658385 21079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:53.658504 21079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:53.658589 21079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:53.691046 21079 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:36353
I20260926 23:16:53.691097 21205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:36353 every 8 connection(s)
I20260926 23:16:53.691892 21079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:16:53.700526 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 21079
I20260926 23:16:53.700721 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:16:53.701182 16633 external_mini_cluster.cc:1428] Setting key 37075a7a39642cf3b9bed04e45719e77
Sep 26 23:16:53 dist-test-slave-vcv1 krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:16:53.711129 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.694472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:46751 (local address 127.16.62.126:41675)
0926 23:16:53.694683 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.694689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.695725 (+ 1036us) server_negotiation.cc:407] Connection header received
0926 23:16:53.697773 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.697779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.697863 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.698007 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:53.699354 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.700074 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.701776 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.702476 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.702637 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.706289 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:53.706293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:53.706296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:53.706325 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:53.708761 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:53.709381 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:53.709385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:53.709387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:53.709489 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:53.709841 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:53.709843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:53.709845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:53.710085 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:53.710192 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.710749 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.710890 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:16:53.712040 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.694923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41675 (local address 127.16.62.67:46751)
0926 23:16:53.695550 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:53.695609 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:53.697502 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:53.698213 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:53.698221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:53.698683 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:53.699191 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.699207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.700247 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.700251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.701619 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.701630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.702689 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.702694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.702871 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.703619 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:16:53.703645 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:53.706115 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:53.708886 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:53.708892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:53.708902 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:53.709288 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:53.709577 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:16:53.709580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:16:53.709582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:16:53.709751 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:16:53.710192 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:53.710197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:53.710456 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 23:16:53.710667 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 23:16:53.711025 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":246,"threads_started":1}
I20260926 23:16:53.713377 21206 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41675
I20260926 23:16:53.713663 21206 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:53.714262 21206 heartbeater.cc:511] Master 127.16.62.126:41675 requested a full tablet report, sending...
I20260926 23:16:53.715683 20753 ts_manager.cc:194] Registered new tserver with Master: a240b945f67c48aeba7a95e69f246acc (127.16.62.67:36353)
I20260926 23:16:53.716737 20753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:46751
I20260926 23:16:53.716921 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:16:53.734211 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.721393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.16.62.126:41675)
0926 23:16:53.721565 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.721572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.722648 (+ 1076us) server_negotiation.cc:407] Connection header received
0926 23:16:53.722704 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.722708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.722787 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.722878 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:53.724764 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.725807 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.726984 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.727643 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.727769 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.729875 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:53.729879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:53.729883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:53.729909 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:53.732062 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:53.732651 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:53.732655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:53.732657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:53.732771 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:53.733059 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:53.733062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:53.733064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:53.733343 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:53.733510 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.733758 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.733893 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:16:53.737877 20753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:16:53.747282 20753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:53.754033 20941 heartbeater.cc:503] Master 127.16.62.126:41675 was elected leader, sending a full tablet report...
I20260926 23:16:53.766683 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.758698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:36353 (local address 127.0.0.1:47310)
0926 23:16:53.759394 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:53.759428 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:53.759587 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:53.760630 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:53.760639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:53.760672 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:53.761092 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.761101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.762695 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.762703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.764105 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.764123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.765757 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.765763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.765981 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.766045 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 23:16:53.766179 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 23:16:53.766358 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":335,"threads_started":1}
I20260926 23:16:53.767602 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.758849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.16.62.67:36353)
0926 23:16:53.759432 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.759438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.759519 (+ 81us) server_negotiation.cc:407] Connection header received
0926 23:16:53.759802 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.759813 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.760078 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.760318 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:53.761299 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.762573 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.764358 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.765561 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.765714 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.765937 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.767095 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.767247 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":286,"threads_started":1}
I20260926 23:16:53.771502 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 37d8af5023e64488ac31dc6a32cc85e6 (DEFAULT_TABLE table=test-table [id=b43d2383806241b8b9bf536c2f706759]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:53.773022 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d8af5023e64488ac31dc6a32cc85e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:53.774587 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.760984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.16.62.66:34803)
0926 23:16:53.766374 (+ 5390us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.766382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.766409 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:16:53.766497 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.766505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.766735 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.766930 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:53.768277 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.769750 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.772191 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.773566 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.773708 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.773885 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.774120 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.774286 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5235,"thread_start_us":5103,"threads_started":1}
I20260926 23:16:53.774641 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.760401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:34803 (local address 127.0.0.1:39960)
0926 23:16:53.760952 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:53.760970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:53.761085 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:53.767179 (+ 6094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:53.767184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:53.767209 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:53.767546 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.767557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.769958 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.769963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.771977 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.771993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.773765 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.773771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.773937 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.773968 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:16:53.774069 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 23:16:53.774205 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":1,"thread_start_us":751,"threads_started":1}
I20260926 23:16:53.775344 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.759775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.0.0.1:41112)
0926 23:16:53.760322 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:53.760339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:53.760439 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:53.769544 (+ 9105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:53.769548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:53.769573 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:53.769920 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.769950 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.771201 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.771207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.772984 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.772994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.774602 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.774611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.774806 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.774841 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:16:53.774901 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 23:16:53.775054 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":270,"threads_started":1}
I20260926 23:16:53.776279 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.760125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.16.62.65:35221)
0926 23:16:53.769011 (+ 8886us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.769019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.769046 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:16:53.769125 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.769132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.769348 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.769536 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:53.770092 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.771068 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.773198 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.774370 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.775578 (+ 1208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.775805 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.775874 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.775999 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8724,"thread_start_us":8559,"threads_started":1}
I20260926 23:16:53.778313 21009 tablet_service.cc:1467] Processing CreateTablet for tablet 37d8af5023e64488ac31dc6a32cc85e6 (DEFAULT_TABLE table=test-table [id=b43d2383806241b8b9bf536c2f706759]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:53.779084 20876 tablet_service.cc:1467] Processing CreateTablet for tablet 37d8af5023e64488ac31dc6a32cc85e6 (DEFAULT_TABLE table=test-table [id=b43d2383806241b8b9bf536c2f706759]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:53.779640 21009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d8af5023e64488ac31dc6a32cc85e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:53.780098 20876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d8af5023e64488ac31dc6a32cc85e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:53.787292 21224 tablet_bootstrap.cc:492] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: Bootstrap starting.
I20260926 23:16:53.790894 21224 tablet_bootstrap.cc:654] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:53.791504 21225 tablet_bootstrap.cc:492] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf: Bootstrap starting.
I20260926 23:16:53.792376 21224 log.cc:824] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:53.794906 21225 tablet_bootstrap.cc:654] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:53.795400 21224 tablet_bootstrap.cc:492] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: No bootstrap required, opened a new log
I20260926 23:16:53.795637 21224 ts_tablet_manager.cc:1397] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 23:16:53.796054 21225 log.cc:824] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:53.798861 21225 tablet_bootstrap.cc:492] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf: No bootstrap required, opened a new log
I20260926 23:16:53.799098 21225 ts_tablet_manager.cc:1397] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 23:16:53.800477 21227 tablet_bootstrap.cc:492] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: Bootstrap starting.
I20260926 23:16:53.803822 21227 tablet_bootstrap.cc:654] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:53.805028 21227 log.cc:824] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:53.805018 21224 raft_consensus.cc:348] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } }
I20260926 23:16:53.805438 21224 raft_consensus.cc:374] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:53.805629 21224 raft_consensus.cc:729] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a240b945f67c48aeba7a95e69f246acc, State: Initialized, Role: FOLLOWER
I20260926 23:16:53.806267 21224 consensus_queue.cc:260] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } }
I20260926 23:16:53.806427 21225 raft_consensus.cc:348] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } }
I20260926 23:16:53.806948 21225 raft_consensus.cc:374] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:53.807076 21225 raft_consensus.cc:729] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea92c4329b646dbaf55add74452cbaf, State: Initialized, Role: FOLLOWER
I20260926 23:16:53.807327 21206 heartbeater.cc:503] Master 127.16.62.126:41675 was elected leader, sending a full tablet report...
I20260926 23:16:53.807755 21225 consensus_queue.cc:260] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } }
I20260926 23:16:53.808112 21224 ts_tablet_manager.cc:1428] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 23:16:53.809762 21227 tablet_bootstrap.cc:492] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: No bootstrap required, opened a new log
I20260926 23:16:53.810138 21227 ts_tablet_manager.cc:1397] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 23:16:53.810351 21230 raft_consensus.cc:482] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:53.812964 21225 ts_tablet_manager.cc:1428] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
I20260926 23:16:53.810715 21230 raft_consensus.cc:504] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } }
I20260926 23:16:53.814965 21230 leader_election.cc:288] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ea92c4329b646dbaf55add74452cbaf (127.16.62.65:35221), 12a4d5ce1d2e4c13bef78cf9e19dbe36 (127.16.62.66:34803)
I20260926 23:16:53.819720 21227 raft_consensus.cc:348] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } }
I20260926 23:16:53.820122 21227 raft_consensus.cc:374] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:53.820261 21227 raft_consensus.cc:729] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a4d5ce1d2e4c13bef78cf9e19dbe36, State: Initialized, Role: FOLLOWER
I20260926 23:16:53.820796 21227 consensus_queue.cc:260] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } }
I20260926 23:16:53.821818 21074 heartbeater.cc:503] Master 127.16.62.126:41675 was elected leader, sending a full tablet report...
I20260926 23:16:53.822472 21227 ts_tablet_manager.cc:1428] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260926 23:16:53.823843 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.815844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.16.62.67:57505)
0926 23:16:53.815995 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:53.816013 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:53.816127 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:53.816587 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:53.816593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:53.816617 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:53.816817 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.816828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.818425 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.818433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.820540 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.820552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.823093 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.823101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.823312 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.823338 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 23:16:53.823454 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 23:16:53.823585 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":25}
I20260926 23:16:53.823844 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.815726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:57505 (local address 127.16.62.65:35221)
0926 23:16:53.815931 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.815937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.816091 (+ 154us) server_negotiation.cc:407] Connection header received
0926 23:16:53.816261 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.816267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.816344 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.816428 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:53.816936 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.818259 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.820670 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.821853 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.823120 (+ 1267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.823201 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.823475 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.823596 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:16:53.824105 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.816635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:46483 (local address 127.16.62.66:34803)
0926 23:16:53.816838 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.816843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.817425 (+ 582us) server_negotiation.cc:407] Connection header received
0926 23:16:53.817668 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.817677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.817755 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.817812 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:16:53.818477 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.819347 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.821770 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.822995 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.823132 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.823201 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.823736 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.823873 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 23:16:53.824764 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.816820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:34803 (local address 127.16.62.67:46483)
0926 23:16:53.817333 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:53.817350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:53.817518 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:53.817972 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:53.817977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:53.817997 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:16:53.818342 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.818353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.819495 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.819500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.821569 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:53.821582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.823333 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.823339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:53.823533 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.823566 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 23:16:53.824257 (+ 691us) client_negotiation.cc:240] Negotiation successful
0926 23:16:53.824392 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":19,"thread_start_us":256,"threads_started":1}
I20260926 23:16:53.824821 20896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8af5023e64488ac31dc6a32cc85e6" candidate_uuid: "a240b945f67c48aeba7a95e69f246acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea92c4329b646dbaf55add74452cbaf" is_pre_election: true
I20260926 23:16:53.825345 20896 raft_consensus.cc:2454] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a240b945f67c48aeba7a95e69f246acc in term 0.
I20260926 23:16:53.826012 21097 leader_election.cc:302] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ea92c4329b646dbaf55add74452cbaf, a240b945f67c48aeba7a95e69f246acc; no voters:
I20260926 23:16:53.826012 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8af5023e64488ac31dc6a32cc85e6" candidate_uuid: "a240b945f67c48aeba7a95e69f246acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" is_pre_election: true
I20260926 23:16:53.826438 21230 raft_consensus.cc:2781] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:53.826429 21029 raft_consensus.cc:2454] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a240b945f67c48aeba7a95e69f246acc in term 0.
I20260926 23:16:53.826581 21230 raft_consensus.cc:482] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:53.826680 21230 raft_consensus.cc:3037] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:53.828898 21230 raft_consensus.cc:504] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } }
I20260926 23:16:53.829367 21230 leader_election.cc:288] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [CANDIDATE]: Term 1 election: Requested vote from peers 9ea92c4329b646dbaf55add74452cbaf (127.16.62.65:35221), 12a4d5ce1d2e4c13bef78cf9e19dbe36 (127.16.62.66:34803)
I20260926 23:16:53.829764 20896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8af5023e64488ac31dc6a32cc85e6" candidate_uuid: "a240b945f67c48aeba7a95e69f246acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea92c4329b646dbaf55add74452cbaf"
I20260926 23:16:53.829844 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d8af5023e64488ac31dc6a32cc85e6" candidate_uuid: "a240b945f67c48aeba7a95e69f246acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36"
I20260926 23:16:53.829952 20896 raft_consensus.cc:3037] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:53.830032 21029 raft_consensus.cc:3037] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:53.832296 20896 raft_consensus.cc:2454] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a240b945f67c48aeba7a95e69f246acc in term 1.
I20260926 23:16:53.832316 21029 raft_consensus.cc:2454] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a240b945f67c48aeba7a95e69f246acc in term 1.
I20260926 23:16:53.832664 21097 leader_election.cc:302] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a4d5ce1d2e4c13bef78cf9e19dbe36, a240b945f67c48aeba7a95e69f246acc; no voters:
I20260926 23:16:53.832952 21230 raft_consensus.cc:2781] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:53.833104 21230 raft_consensus.cc:686] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 1 LEADER]: Becoming Leader. State: Replica: a240b945f67c48aeba7a95e69f246acc, State: Running, Role: LEADER
I20260926 23:16:53.833633 21230 consensus_queue.cc:237] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } }
I20260926 23:16:53.838060 20752 catalog_manager.cc:5495] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc reported cstate change: term changed from 0 to 1, leader changed from <none> to a240b945f67c48aeba7a95e69f246acc (127.16.62.67). New cstate: current_term: 1 leader_uuid: "a240b945f67c48aeba7a95e69f246acc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a240b945f67c48aeba7a95e69f246acc" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 36353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 } health_report { overall_health: UNKNOWN } } }
I20260926 23:16:53.869283 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.864270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.16.62.67:36353)
0926 23:16:53.864495 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.864500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.864514 (+ 14us) server_negotiation.cc:407] Connection header received
0926 23:16:53.864590 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.864594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.864662 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.864815 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:16:53.865163 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.865886 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.867184 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.867567 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.867693 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.868200 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:16:53.868684 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:16:53.868844 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.868944 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.869045 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:16:53.876879 20896 raft_consensus.cc:1261] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [term 1 FOLLOWER]: Refusing update from remote peer a240b945f67c48aeba7a95e69f246acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:53.876921 21029 raft_consensus.cc:1261] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [term 1 FOLLOWER]: Refusing update from remote peer a240b945f67c48aeba7a95e69f246acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:53.877821 21231 consensus_queue.cc:1035] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a4d5ce1d2e4c13bef78cf9e19dbe36" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:53.878163 21230 consensus_queue.cc:1035] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea92c4329b646dbaf55add74452cbaf" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:53.903400 20752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:53.903698 20752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:53.907322 20752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1eba38fffa74a1195763118c3d7aa05: Sending DeleteTablet for 3 replicas of tablet 37d8af5023e64488ac31dc6a32cc85e6
I20260926 23:16:53.908599 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d8af5023e64488ac31dc6a32cc85e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:53 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:47310
I20260926 23:16:53.908592 20876 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d8af5023e64488ac31dc6a32cc85e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:53 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:41112
I20260926 23:16:53.908886 21009 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d8af5023e64488ac31dc6a32cc85e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:53 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:39960
I20260926 23:16:53.910550 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 20814
I20260926 23:16:53.921039 21247 tablet_replica.cc:330] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: stopping tablet replica
I20260926 23:16:53.921551 21247 raft_consensus.cc:2229] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:16:53.922335 21247 raft_consensus.cc:2258] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:53.923134 21249 tablet_replica.cc:330] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: stopping tablet replica
I20260926 23:16:53.923594 21249 raft_consensus.cc:2229] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:53.924165 21249 raft_consensus.cc:2258] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:53.927980 21249 ts_tablet_manager.cc:1905] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:53.929872 21247 ts_tablet_manager.cc:1905] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:53.931248 21248 tablet_replica.cc:330] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf: stopping tablet replica
I20260926 23:16:53.932523 21248 raft_consensus.cc:2229] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:53.934983 21249 ts_tablet_manager.cc:1918] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:16:53.935207 21249 log.cc:1196] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/37d8af5023e64488ac31dc6a32cc85e6
I20260926 23:16:53.935853 21249 ts_tablet_manager.cc:1939] T 37d8af5023e64488ac31dc6a32cc85e6 P 12a4d5ce1d2e4c13bef78cf9e19dbe36: Deleting consensus metadata
Sep 26 23:16:53 dist-test-slave-vcv1 krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:16:53.937858 20740 catalog_manager.cc:4841] TS 12a4d5ce1d2e4c13bef78cf9e19dbe36 (127.16.62.66:34803): tablet 37d8af5023e64488ac31dc6a32cc85e6 (table test-table [id=b43d2383806241b8b9bf536c2f706759]) successfully deleted
I20260926 23:16:53.938545 21247 ts_tablet_manager.cc:1918] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:16:53.938755 21247 log.cc:1196] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/37d8af5023e64488ac31dc6a32cc85e6
I20260926 23:16:53.938762 21248 raft_consensus.cc:2258] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:53.939353 21247 ts_tablet_manager.cc:1939] T 37d8af5023e64488ac31dc6a32cc85e6 P a240b945f67c48aeba7a95e69f246acc: Deleting consensus metadata
I20260926 23:16:53.940974 20740 catalog_manager.cc:4841] TS a240b945f67c48aeba7a95e69f246acc (127.16.62.67:36353): tablet 37d8af5023e64488ac31dc6a32cc85e6 (table test-table [id=b43d2383806241b8b9bf536c2f706759]) successfully deleted
I20260926 23:16:53.941464 21248 ts_tablet_manager.cc:1905] T 37d8af5023e64488ac31dc6a32cc85e6 P 9ea92c4329b646dbaf55add74452cbaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:53.945647 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:53.912383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.16.62.65:35221)
0926 23:16:53.912601 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:53.912609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:53.926890 (+ 14281us) server_negotiation.cc:407] Connection header received
0926 23:16:53.927320 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:53.927326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:53.927412 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:53.927524 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:53.929108 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.930241 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.932061 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:53.932767 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:53.932870 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:53.938307 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:53.938323 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:53.938342 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:53.938380 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:53.942208 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:53.943091 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:53.943099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:53.943102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:53.943261 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:53.943682 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:53.943687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:53.943690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:53.944216 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:53.944516 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:53.944812 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 23:16:53.945050 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:16:53.946406 20936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:54.033738 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 20814
W20260926 23:16:54.055759 20740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.62.65:35221 (error 108)
I20260926 23:16:54.056578 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 20947
I20260926 23:16:54.057775 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.057207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.0.0.1:41116)
0926 23:16:54.057505 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:54.057613 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260926 23:16:54.057991 20740 catalog_manager.cc:4584] TS 9ea92c4329b646dbaf55add74452cbaf (127.16.62.65:35221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37d8af5023e64488ac31dc6a32cc85e6: Network error: Client connection negotiation failed: client connection to 127.16.62.65:35221: connect: Connection refused (error 111)
Sep 26 23:16:54 dist-test-slave-vcv1 krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:16:54.071523 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.057400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.16.62.66:34803)
0926 23:16:54.057575 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:54.057582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:54.057606 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:16:54.057851 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:54.057858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:54.057939 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:54.058058 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:54.059357 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:54.060101 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:54.061527 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:54.061910 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:54.062036 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:54.065979 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:54.065992 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:54.066006 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:54.066041 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:54.068883 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:54.069523 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:54.069530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:54.069532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:54.069654 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:54.070014 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:54.070020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:54.070023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:54.070375 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:54.070661 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:54.070973 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 23:16:54.071191 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:16:54.072227 21069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:54.111928 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.111409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.0.0.1:41120)
0926 23:16:54.111632 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:54.111745 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 23:16:54.113161 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.112707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.0.0.1:41134)
0926 23:16:54.112880 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:54.112983 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 23:16:54.167716 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 20947
I20260926 23:16:54.173696 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.173213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.0.0.1:41136)
0926 23:16:54.173399 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:54.173495 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 23:16:54.175030 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.174663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.0.0.1:41148)
0926 23:16:54.174800 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:54.174878 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 23:16:54.190409 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 21079
Sep 26 23:16:54 dist-test-slave-vcv1 krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:16:54.207299 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.191148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.16.62.67:36353)
0926 23:16:54.191321 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:54.191328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:54.191397 (+ 69us) server_negotiation.cc:407] Connection header received
0926 23:16:54.191620 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:54.191626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:54.191733 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:54.191851 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:54.193144 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:54.194209 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:54.195457 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:54.195966 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:54.196098 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:54.201382 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:54.201392 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:54.201407 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:16:54.201440 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:54.204548 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:54.205153 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:54.205161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:54.205165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:54.205382 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:16:54.205776 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:16:54.205784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:16:54.205787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:16:54.206136 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:54.206457 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:54.206734 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 23:16:54.206939 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:16:54.208001 21201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:54.255520 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.254973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.0.0.1:41156)
0926 23:16:54.255269 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:54.255361 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260926 23:16:54.256690 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.256300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.0.0.1:41164)
0926 23:16:54.256512 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:54.256586 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 23:16:54.328754 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 21079
I20260926 23:16:54.347822 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 20722
I20260926 23:16:54.349146 20783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:54.455986 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.455253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35221 (local address 127.0.0.1:41180)
0926 23:16:54.455490 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:54.455617 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 23:16:54.456238 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 20722
2026-09-26T23:16:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3996 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T23:16:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:54Z Disabled control of system clock
I20260926 23:16:54.517627 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:43461
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:43461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 23:16:54.827553 21259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:54.827805 21259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:54.827939 21259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:54.835326 21259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:54.835433 21259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:54.835469 21259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:54.835517 21259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:54.835603 21259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:16:54.844600 21259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:43461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:43461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:54.845860 21259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:54.847656 21259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:54.859462 21264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:54.859547 21267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:54.859596 21265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:54.860287 21259 server_base.cc:1034] running on GCE node
I20260926 23:16:54.861080 21259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:54.862442 21259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:54.863662 21259 hybrid_clock.cc:648] HybridClock initialized: now 1790464614863634 us; error 56 us; skew 500 ppm
I20260926 23:16:54.863996 21259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:16:54.866442 21259 webserver.cc:460] Webserver started at http://127.16.62.126:34877/ using document root <none> and password file <none>
I20260926 23:16:54.867138 21259 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:54.867269 21259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:54.867585 21259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:54.870404 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "1ed416e90ce0488c86724c47efb6b874"
format_stamp: "Formatted at 2026-09-26 23:16:54 on dist-test-slave-vcv1"
server_key: "f7b8ce310f2d15d00fafdf5cade58119"
server_key_iv: "711b30f4f97d860cc00036efd681b2de"
server_key_version: "encryptionkey@0"
I20260926 23:16:54.871173 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "1ed416e90ce0488c86724c47efb6b874"
format_stamp: "Formatted at 2026-09-26 23:16:54 on dist-test-slave-vcv1"
server_key: "f7b8ce310f2d15d00fafdf5cade58119"
server_key_iv: "711b30f4f97d860cc00036efd681b2de"
server_key_version: "encryptionkey@0"
I20260926 23:16:54.876062 21259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 23:16:54.879884 21273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:54.881264 21259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 23:16:54.881460 21259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "1ed416e90ce0488c86724c47efb6b874"
format_stamp: "Formatted at 2026-09-26 23:16:54 on dist-test-slave-vcv1"
server_key: "f7b8ce310f2d15d00fafdf5cade58119"
server_key_iv: "711b30f4f97d860cc00036efd681b2de"
server_key_version: "encryptionkey@0"
I20260926 23:16:54.881677 21259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:54.887296 21259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:54.888031 21259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:54.888393 21259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:54.906145 21259 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:43461
I20260926 23:16:54.906176 21324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:43461 every 8 connection(s)
I20260926 23:16:54.907454 21259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:54.910983 21325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:54.913090 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 21259
I20260926 23:16:54.913300 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:54.913923 16633 external_mini_cluster.cc:1428] Setting key dd92e41b25073ffa2585f57687cfab33
I20260926 23:16:54.923256 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:54.915979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.16.62.126:43461)
0926 23:16:54.917045 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:54.917059 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:54.917104 (+ 45us) server_negotiation.cc:407] Connection header received
0926 23:16:54.919466 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:54.919496 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:54.920030 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:54.920456 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:54.921435 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:54.921444 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:54.921459 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:54.921475 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:54.921553 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:54.921572 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:54.921806 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:54.921936 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 23:16:54.922163 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":376,"threads_started":1}
I20260926 23:16:54.923902 21325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874: Bootstrap starting.
I20260926 23:16:54.926718 21325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:54.927641 21325 log.cc:824] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:54.929996 21325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874: No bootstrap required, opened a new log
I20260926 23:16:54.936623 21325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed416e90ce0488c86724c47efb6b874" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43461 } }
I20260926 23:16:54.936932 21325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:54.937057 21325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed416e90ce0488c86724c47efb6b874, State: Initialized, Role: FOLLOWER
I20260926 23:16:54.937650 21325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed416e90ce0488c86724c47efb6b874" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43461 } }
I20260926 23:16:54.937872 21325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:54.938019 21325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:54.938186 21325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:54.940212 21325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed416e90ce0488c86724c47efb6b874" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43461 } }
I20260926 23:16:54.940685 21325 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed416e90ce0488c86724c47efb6b874; no voters:
I20260926 23:16:54.941270 21325 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:54.941407 21330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:54.942075 21330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed416e90ce0488c86724c47efb6b874, State: Running, Role: LEADER
I20260926 23:16:54.942642 21330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed416e90ce0488c86724c47efb6b874" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43461 } }
I20260926 23:16:54.943176 21325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:54.945281 21331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed416e90ce0488c86724c47efb6b874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed416e90ce0488c86724c47efb6b874" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43461 } } }
I20260926 23:16:54.945644 21331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:54.945374 21332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed416e90ce0488c86724c47efb6b874. Latest consensus state: current_term: 1 leader_uuid: "1ed416e90ce0488c86724c47efb6b874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed416e90ce0488c86724c47efb6b874" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43461 } } }
I20260926 23:16:54.946002 21332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:54.947255 21336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:54.952281 21336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:54.959908 21336 catalog_manager.cc:1347] Generated new cluster ID: 01a07ab33062420bb272e54aae5a79f6
I20260926 23:16:54.960006 21336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:55.006330 21336 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:55.006750 21336 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:55.011564 21336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874: Generated new TSK 0
I20260926 23:16:55.012358 21336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:16:55.018059 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:43461
--builtin_ntp_servers=127.16.62.84:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 23:16:55.345402 21349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:55.345752 21349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:55.345920 21349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:55.353801 21349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:55.353955 21349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:55.354161 21349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:16:55.363231 21349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:43461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:55.364689 21349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:55.366544 21349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:55.379807 21354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:55.379894 21355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:55.379856 21357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:55.380755 21349 server_base.cc:1034] running on GCE node
I20260926 23:16:55.381326 21349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:55.382252 21349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:55.383508 21349 hybrid_clock.cc:648] HybridClock initialized: now 1790464615383450 us; error 35 us; skew 500 ppm
I20260926 23:16:55.383832 21349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:16:55.386162 21349 webserver.cc:460] Webserver started at http://127.16.62.65:39479/ using document root <none> and password file <none>
I20260926 23:16:55.386816 21349 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:55.386955 21349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:55.387300 21349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:55.390159 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "295003248359433f92b8afad8501e63d"
format_stamp: "Formatted at 2026-09-26 23:16:55 on dist-test-slave-vcv1"
server_key: "4c23cc33d46e5cfb8dfff6f380adb1cb"
server_key_iv: "4ed0d695d4a289f62c28a788f3312dc6"
server_key_version: "encryptionkey@0"
I20260926 23:16:55.390982 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "295003248359433f92b8afad8501e63d"
format_stamp: "Formatted at 2026-09-26 23:16:55 on dist-test-slave-vcv1"
server_key: "4c23cc33d46e5cfb8dfff6f380adb1cb"
server_key_iv: "4ed0d695d4a289f62c28a788f3312dc6"
server_key_version: "encryptionkey@0"
I20260926 23:16:55.396112 21349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 23:16:55.399443 21363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:55.400743 21349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:55.400918 21349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "295003248359433f92b8afad8501e63d"
format_stamp: "Formatted at 2026-09-26 23:16:55 on dist-test-slave-vcv1"
server_key: "4c23cc33d46e5cfb8dfff6f380adb1cb"
server_key_iv: "4ed0d695d4a289f62c28a788f3312dc6"
server_key_version: "encryptionkey@0"
I20260926 23:16:55.401141 21349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:55.406491 21349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:55.407250 21349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:55.407572 21349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:55.408545 21349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:55.409926 21349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:55.410054 21349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:55.410194 21349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:55.410287 21349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:55.444198 21349 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:46023
I20260926 23:16:55.444257 21475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:46023 every 8 connection(s)
I20260926 23:16:55.445567 21349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:16:55.450357 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 21349
I20260926 23:16:55.450618 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:16:55.451296 16633 external_mini_cluster.cc:1428] Setting key 6609e619fe4476d1a7d5dcd9aa879be1
I20260926 23:16:55.454066 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:55.448246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:34543 (local address 127.16.62.126:43461)
0926 23:16:55.448662 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:55.448667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:55.449970 (+ 1303us) server_negotiation.cc:407] Connection header received
0926 23:16:55.451744 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:55.451749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:55.451817 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:55.451942 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:55.453337 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:55.453343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:55.453348 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:55.453367 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:55.453413 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:55.453423 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:55.453544 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:55.453731 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 23:16:55.453803 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":215,"threads_started":1}
I20260926 23:16:55.454357 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:43461
--builtin_ntp_servers=127.16.62.84:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 23:16:55.455076 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:55.448912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:43461 (local address 127.16.62.65:34543)
0926 23:16:55.449829 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:55.449867 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:55.451496 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:55.452224 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:55.452232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:55.452321 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:55.453024 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:16:55.453030 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:55.453080 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:16:55.453190 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:55.453513 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:55.453524 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:55.453533 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:16:55.453817 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 23:16:55.454063 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":338,"threads_started":1}
I20260926 23:16:55.457589 21476 heartbeater.cc:347] Connected to a master server at 127.16.62.126:43461
I20260926 23:16:55.457823 21476 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:55.458297 21476 heartbeater.cc:511] Master 127.16.62.126:43461 requested a full tablet report, sending...
I20260926 23:16:55.460366 21290 ts_manager.cc:194] Registered new tserver with Master: 295003248359433f92b8afad8501e63d (127.16.62.65:46023)
W20260926 23:16:55.771941 21480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:55.772328 21480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:55.772491 21480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:55.779587 21480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:55.779677 21480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:55.779812 21480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:16:55.789011 21480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:43461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:55.790253 21480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:55.792061 21480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:55.804813 21486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:55.805011 21485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:55.805239 21488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:55.805761 21480 server_base.cc:1034] running on GCE node
I20260926 23:16:55.806342 21480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:55.807299 21480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:55.808485 21480 hybrid_clock.cc:648] HybridClock initialized: now 1790464615808443 us; error 39 us; skew 500 ppm
I20260926 23:16:55.808785 21480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:16:55.810878 21480 webserver.cc:460] Webserver started at http://127.16.62.66:37403/ using document root <none> and password file <none>
I20260926 23:16:55.811515 21480 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:55.811627 21480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:55.811955 21480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:55.814685 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "e170a5bfa8544a3db8cb37167ffbe44a"
format_stamp: "Formatted at 2026-09-26 23:16:55 on dist-test-slave-vcv1"
server_key: "bfd0cfdac48055f7b5cbb97122aabf4c"
server_key_iv: "73c2d339137304c034a3241b17f66522"
server_key_version: "encryptionkey@0"
I20260926 23:16:55.815445 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "e170a5bfa8544a3db8cb37167ffbe44a"
format_stamp: "Formatted at 2026-09-26 23:16:55 on dist-test-slave-vcv1"
server_key: "bfd0cfdac48055f7b5cbb97122aabf4c"
server_key_iv: "73c2d339137304c034a3241b17f66522"
server_key_version: "encryptionkey@0"
I20260926 23:16:55.820294 21480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 23:16:55.823585 21494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:55.824873 21480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:55.825040 21480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "e170a5bfa8544a3db8cb37167ffbe44a"
format_stamp: "Formatted at 2026-09-26 23:16:55 on dist-test-slave-vcv1"
server_key: "bfd0cfdac48055f7b5cbb97122aabf4c"
server_key_iv: "73c2d339137304c034a3241b17f66522"
server_key_version: "encryptionkey@0"
I20260926 23:16:55.825245 21480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:55.830555 21480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:55.831285 21480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:55.831543 21480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:55.832430 21480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:55.833683 21480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:55.833806 21480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:55.833920 21480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:55.833997 21480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:55.865356 21480 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:36917
I20260926 23:16:55.865435 21606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:36917 every 8 connection(s)
I20260926 23:16:55.866577 21480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:16:55.872853 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 21480
I20260926 23:16:55.873034 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:16:55.873605 16633 external_mini_cluster.cc:1428] Setting key 95fae5f0eeaa7fdd9fe1935b08809566
I20260926 23:16:55.874419 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:55.869021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:44487 (local address 127.16.62.126:43461)
0926 23:16:55.869236 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:55.869242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:55.870495 (+ 1253us) server_negotiation.cc:407] Connection header received
0926 23:16:55.872017 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:55.872021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:55.872084 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:55.872228 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:55.873523 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:55.873542 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:55.873546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:55.873574 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:55.873633 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:55.873651 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:55.873774 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:55.874066 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 23:16:55.874116 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:16:55.875310 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:55.869528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:43461 (local address 127.16.62.66:44487)
0926 23:16:55.870359 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:55.870390 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:55.871797 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:55.872410 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:55.872417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:55.872497 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:55.873245 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:16:55.873248 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:55.873283 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:16:55.873362 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:55.873832 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:55.873842 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:55.873849 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 23:16:55.874117 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 23:16:55.874341 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":252,"threads_started":1}
I20260926 23:16:55.876595 21607 heartbeater.cc:347] Connected to a master server at 127.16.62.126:43461
I20260926 23:16:55.876575 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:43461
--builtin_ntp_servers=127.16.62.84:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 23:16:55.876886 21607 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:55.877557 21607 heartbeater.cc:511] Master 127.16.62.126:43461 requested a full tablet report, sending...
I20260926 23:16:55.879041 21290 ts_manager.cc:194] Registered new tserver with Master: e170a5bfa8544a3db8cb37167ffbe44a (127.16.62.66:36917)
W20260926 23:16:56.187798 21610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:56.188037 21610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:56.188251 21610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:56.195241 21610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:56.195328 21610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:56.195505 21610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:16:56.204586 21610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:43461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:56.205873 21610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:56.207579 21610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:56.220281 21615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:56.220350 21616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:56.220453 21618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:56.221601 21610 server_base.cc:1034] running on GCE node
I20260926 23:16:56.222116 21610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:56.223030 21610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:56.224262 21610 hybrid_clock.cc:648] HybridClock initialized: now 1790464616224214 us; error 46 us; skew 500 ppm
I20260926 23:16:56.224571 21610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:16:56.226667 21610 webserver.cc:460] Webserver started at http://127.16.62.67:38703/ using document root <none> and password file <none>
I20260926 23:16:56.227325 21610 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:56.227432 21610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:56.227752 21610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:56.230767 21610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "bf2731dbbd6145d68e352592c260be29"
format_stamp: "Formatted at 2026-09-26 23:16:56 on dist-test-slave-vcv1"
server_key: "736129d9e6396d4df1b7e1438cbdaf45"
server_key_iv: "2c74e9d8fa865a521b39136536c7d942"
server_key_version: "encryptionkey@0"
I20260926 23:16:56.231537 21610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "bf2731dbbd6145d68e352592c260be29"
format_stamp: "Formatted at 2026-09-26 23:16:56 on dist-test-slave-vcv1"
server_key: "736129d9e6396d4df1b7e1438cbdaf45"
server_key_iv: "2c74e9d8fa865a521b39136536c7d942"
server_key_version: "encryptionkey@0"
I20260926 23:16:56.236455 21610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:16:56.239624 21624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:56.240918 21610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:16:56.241063 21610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "bf2731dbbd6145d68e352592c260be29"
format_stamp: "Formatted at 2026-09-26 23:16:56 on dist-test-slave-vcv1"
server_key: "736129d9e6396d4df1b7e1438cbdaf45"
server_key_iv: "2c74e9d8fa865a521b39136536c7d942"
server_key_version: "encryptionkey@0"
I20260926 23:16:56.241276 21610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:56.246868 21610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:56.247638 21610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:56.247910 21610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:56.248904 21610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:16:56.250337 21610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:16:56.250440 21610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:56.250569 21610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:16:56.250660 21610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:56.282770 21610 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:43641
I20260926 23:16:56.282822 21736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:43641 every 8 connection(s)
I20260926 23:16:56.284065 21610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:16:56.284813 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 21610
I20260926 23:16:56.285025 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:16:56.285646 16633 external_mini_cluster.cc:1428] Setting key 594b03f3cc134767db9dcb69a697856f
I20260926 23:16:56.292729 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.287005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:53131 (local address 127.16.62.126:43461)
0926 23:16:56.287181 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.287189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.288894 (+ 1705us) server_negotiation.cc:407] Connection header received
0926 23:16:56.290584 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.290589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.290655 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.290744 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.291986 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.291992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.291997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.292015 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.292055 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.292066 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.292199 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.292397 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.292464 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:16:56.293619 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.287708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:43461 (local address 127.16.62.67:53131)
0926 23:16:56.288722 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.288786 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:56.290335 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:56.290968 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:56.290976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:56.291064 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:56.291699 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:16:56.291702 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:56.291746 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:16:56.291827 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:56.292174 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:56.292184 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:56.292192 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:16:56.292465 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 23:16:56.292710 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":350,"threads_started":1}
I20260926 23:16:56.294919 21737 heartbeater.cc:347] Connected to a master server at 127.16.62.126:43461
I20260926 23:16:56.295161 21737 heartbeater.cc:464] Registering TS with master...
I20260926 23:16:56.295696 21737 heartbeater.cc:511] Master 127.16.62.126:43461 requested a full tablet report, sending...
I20260926 23:16:56.297080 21290 ts_manager.cc:194] Registered new tserver with Master: bf2731dbbd6145d68e352592c260be29 (127.16.62.67:43641)
I20260926 23:16:56.301352 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:16:56.308784 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.306080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.16.62.126:43461)
0926 23:16:56.306291 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.306298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.306724 (+ 426us) server_negotiation.cc:407] Connection header received
0926 23:16:56.306937 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.306943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.307005 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.307125 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.308048 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.308054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.308059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.308089 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.308123 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.308179 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.308305 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.308424 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.308486 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 23:16:56.312708 21290 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:16:56.319603 21290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:16:56.337651 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.331777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.16.62.65:46023)
0926 23:16:56.332447 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.332455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.332964 (+ 509us) server_negotiation.cc:407] Connection header received
0926 23:16:56.333292 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.333301 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.333612 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.333793 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.336434 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.336443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.336458 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.336492 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.336564 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.336591 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.336717 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.337063 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.337182 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":385,"threads_started":1}
I20260926 23:16:56.338641 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.331606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:46023 (local address 127.0.0.1:35536)
0926 23:16:56.332824 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.332867 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:56.333064 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:56.334984 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:56.334995 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:56.335116 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:56.336056 (+ 940us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:16:56.336062 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:56.336101 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:16:56.336250 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:56.336825 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:56.336840 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:56.336850 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:16:56.337959 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0926 23:16:56.338069 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":883,"threads_started":1}
I20260926 23:16:56.342886 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.332130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36917 (local address 127.0.0.1:37526)
0926 23:16:56.338902 (+ 6772us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.338925 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:56.339009 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:56.340806 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:56.340813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:56.340874 (+ 61us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:56.341583 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:16:56.341589 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:56.341610 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:16:56.341669 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:56.342292 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:56.342300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:56.342308 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:16:56.342424 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 23:16:56.342486 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6619,"threads_started":1}
I20260926 23:16:56.343755 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.332638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.16.62.66:36917)
0926 23:16:56.333142 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.333149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.339123 (+ 5974us) server_negotiation.cc:407] Connection header received
0926 23:16:56.339208 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.339217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.339449 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.339587 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.341872 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.341884 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.341903 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.341937 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.342009 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.342034 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.342216 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.343326 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.343422 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":263,"threads_started":1}
I20260926 23:16:56.349385 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.333277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:43641 (local address 127.0.0.1:58120)
0926 23:16:56.343070 (+ 9793us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.343088 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:56.343162 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:56.346048 (+ 2886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:56.346055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:56.346089 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:56.346837 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:16:56.346843 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:56.346863 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:16:56.346910 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:56.348764 (+ 1854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:56.348772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:56.348780 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:16:56.348904 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 23:16:56.348966 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9713,"threads_started":1}
I20260926 23:16:56.350256 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.333717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.16.62.67:43641)
0926 23:16:56.345422 (+ 11705us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.345435 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.345479 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:16:56.345561 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.345573 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.345829 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.346012 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.348280 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.348291 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.348308 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.348330 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.348473 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.348499 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.348698 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.349615 (+ 917us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.349727 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11477,"thread_start_us":13363,"threads_started":1}
I20260926 23:16:56.350755 21542 tablet_service.cc:1467] Processing CreateTablet for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef (DEFAULT_TABLE table=test-table [id=932b08c2077548bda12ad2ed89d080a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:56.352088 21542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:56.354048 21411 tablet_service.cc:1467] Processing CreateTablet for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef (DEFAULT_TABLE table=test-table [id=932b08c2077548bda12ad2ed89d080a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:56.354048 21672 tablet_service.cc:1467] Processing CreateTablet for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef (DEFAULT_TABLE table=test-table [id=932b08c2077548bda12ad2ed89d080a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:16:56.355322 21672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:56.355325 21411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:56.362883 21755 tablet_bootstrap.cc:492] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: Bootstrap starting.
I20260926 23:16:56.366221 21755 tablet_bootstrap.cc:654] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:56.367165 21755 log.cc:824] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:56.369247 21757 tablet_bootstrap.cc:492] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: Bootstrap starting.
I20260926 23:16:56.370033 21758 tablet_bootstrap.cc:492] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: Bootstrap starting.
I20260926 23:16:56.371172 21755 tablet_bootstrap.cc:492] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: No bootstrap required, opened a new log
I20260926 23:16:56.371439 21755 ts_tablet_manager.cc:1397] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 23:16:56.372905 21757 tablet_bootstrap.cc:654] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:56.373448 21758 tablet_bootstrap.cc:654] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:56.373888 21757 log.cc:824] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:56.374528 21758 log.cc:824] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:56.376612 21757 tablet_bootstrap.cc:492] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: No bootstrap required, opened a new log
I20260926 23:16:56.376847 21757 ts_tablet_manager.cc:1397] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260926 23:16:56.377322 21758 tablet_bootstrap.cc:492] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: No bootstrap required, opened a new log
I20260926 23:16:56.377545 21758 ts_tablet_manager.cc:1397] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:16:56.381757 21755 raft_consensus.cc:348] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } }
I20260926 23:16:56.382146 21755 raft_consensus.cc:374] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:56.382244 21755 raft_consensus.cc:729] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e170a5bfa8544a3db8cb37167ffbe44a, State: Initialized, Role: FOLLOWER
I20260926 23:16:56.382874 21755 consensus_queue.cc:260] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } }
I20260926 23:16:56.383865 21607 heartbeater.cc:503] Master 127.16.62.126:43461 was elected leader, sending a full tablet report...
I20260926 23:16:56.384421 21757 raft_consensus.cc:348] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } }
I20260926 23:16:56.384739 21757 raft_consensus.cc:374] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:56.384848 21757 raft_consensus.cc:729] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295003248359433f92b8afad8501e63d, State: Initialized, Role: FOLLOWER
I20260926 23:16:56.385421 21757 consensus_queue.cc:260] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } }
I20260926 23:16:56.386204 21755 ts_tablet_manager.cc:1428] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260926 23:16:56.387157 21476 heartbeater.cc:503] Master 127.16.62.126:43461 was elected leader, sending a full tablet report...
I20260926 23:16:56.387369 21758 raft_consensus.cc:348] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } }
I20260926 23:16:56.387750 21758 raft_consensus.cc:374] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:56.387852 21757 ts_tablet_manager.cc:1428] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260926 23:16:56.387862 21758 raft_consensus.cc:729] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2731dbbd6145d68e352592c260be29, State: Initialized, Role: FOLLOWER
I20260926 23:16:56.388499 21758 consensus_queue.cc:260] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } }
I20260926 23:16:56.390271 21737 heartbeater.cc:503] Master 127.16.62.126:43461 was elected leader, sending a full tablet report...
I20260926 23:16:56.390861 21758 ts_tablet_manager.cc:1428] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 23:16:56.415889 21762 raft_consensus.cc:482] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:16:56.416239 21762 raft_consensus.cc:504] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } }
I20260926 23:16:56.417755 21762 leader_election.cc:288] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2731dbbd6145d68e352592c260be29 (127.16.62.67:43641), e170a5bfa8544a3db8cb37167ffbe44a (127.16.62.66:36917)
I20260926 23:16:56.420941 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.418159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:45269 (local address 127.16.62.66:36917)
0926 23:16:56.418362 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.418372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.418885 (+ 513us) server_negotiation.cc:407] Connection header received
0926 23:16:56.419046 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.419052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.419106 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.419201 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.420324 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.420329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.420334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.420352 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.420387 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.420402 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.420535 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.420646 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.420693 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 23:16:56.420941 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.418179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:51857 (local address 127.16.62.67:43641)
0926 23:16:56.418365 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.418373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.418957 (+ 584us) server_negotiation.cc:407] Connection header received
0926 23:16:56.419049 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.419054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.419106 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.419201 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.420324 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.420329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.420334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.420352 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.420387 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.420402 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.420535 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.420634 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.420691 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:16:56.421262 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.418103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:43641 (local address 127.16.62.65:51857)
0926 23:16:56.418862 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.418876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:56.418952 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:56.419243 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:56.419248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:56.419272 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:56.420058 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:16:56.420063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:56.420109 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:16:56.420179 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:56.420513 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:56.420518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:56.420523 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:16:56.420706 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 23:16:56.420784 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":21,"thread_start_us":434,"threads_started":1}
I20260926 23:16:56.421535 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.418004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36917 (local address 127.16.62.65:45269)
0926 23:16:56.418776 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.418796 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:56.418923 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:56.419243 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:56.419248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:56.419272 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:16:56.420114 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:16:56.420116 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:16:56.420127 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:16:56.420179 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:16:56.420513 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:16:56.420518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:16:56.420523 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:16:56.420713 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 23:16:56.420771 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":378,"threads_started":1}
I20260926 23:16:56.422238 21692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e22f4a10ba4f0a8aedcc78a7d944ef" candidate_uuid: "295003248359433f92b8afad8501e63d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2731dbbd6145d68e352592c260be29" is_pre_election: true
I20260926 23:16:56.422328 21562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e22f4a10ba4f0a8aedcc78a7d944ef" candidate_uuid: "295003248359433f92b8afad8501e63d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" is_pre_election: true
I20260926 23:16:56.422680 21692 raft_consensus.cc:2454] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 295003248359433f92b8afad8501e63d in term 0.
I20260926 23:16:56.422761 21562 raft_consensus.cc:2454] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 295003248359433f92b8afad8501e63d in term 0.
I20260926 23:16:56.423390 21366 leader_election.cc:302] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 295003248359433f92b8afad8501e63d, bf2731dbbd6145d68e352592c260be29, e170a5bfa8544a3db8cb37167ffbe44a; no voters:
I20260926 23:16:56.423744 21762 raft_consensus.cc:2781] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:16:56.423877 21762 raft_consensus.cc:482] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:16:56.423996 21762 raft_consensus.cc:3037] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:56.426167 21762 raft_consensus.cc:504] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } }
I20260926 23:16:56.426718 21762 leader_election.cc:288] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [CANDIDATE]: Term 1 election: Requested vote from peers bf2731dbbd6145d68e352592c260be29 (127.16.62.67:43641), e170a5bfa8544a3db8cb37167ffbe44a (127.16.62.66:36917)
I20260926 23:16:56.427081 21692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e22f4a10ba4f0a8aedcc78a7d944ef" candidate_uuid: "295003248359433f92b8afad8501e63d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2731dbbd6145d68e352592c260be29"
I20260926 23:16:56.427137 21562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e22f4a10ba4f0a8aedcc78a7d944ef" candidate_uuid: "295003248359433f92b8afad8501e63d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e170a5bfa8544a3db8cb37167ffbe44a"
I20260926 23:16:56.427295 21562 raft_consensus.cc:3037] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:56.427286 21692 raft_consensus.cc:3037] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:56.430438 21562 raft_consensus.cc:2454] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 295003248359433f92b8afad8501e63d in term 1.
I20260926 23:16:56.430438 21692 raft_consensus.cc:2454] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 295003248359433f92b8afad8501e63d in term 1.
I20260926 23:16:56.430812 21365 leader_election.cc:302] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 295003248359433f92b8afad8501e63d, bf2731dbbd6145d68e352592c260be29; no voters:
I20260926 23:16:56.431121 21762 raft_consensus.cc:2781] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:56.431550 21762 raft_consensus.cc:686] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 1 LEADER]: Becoming Leader. State: Replica: 295003248359433f92b8afad8501e63d, State: Running, Role: LEADER
I20260926 23:16:56.431960 21762 consensus_queue.cc:237] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } }
I20260926 23:16:56.436388 21290 catalog_manager.cc:5495] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d reported cstate change: term changed from 0 to 1, leader changed from <none> to 295003248359433f92b8afad8501e63d (127.16.62.65). New cstate: current_term: 1 leader_uuid: "295003248359433f92b8afad8501e63d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295003248359433f92b8afad8501e63d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 46023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 } health_report { overall_health: UNKNOWN } } }
W20260926 23:16:56.447825 21477 tablet.cc:2369] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:16:56.461336 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.459079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.16.62.65:46023)
0926 23:16:56.459251 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.459260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.459351 (+ 91us) server_negotiation.cc:407] Connection header received
0926 23:16:56.459609 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.459614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.459694 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.459835 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.460694 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.460700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.460704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.460722 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.460762 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.460775 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.460894 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.460988 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.461048 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:16:56.468690 21562 raft_consensus.cc:1261] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [term 1 FOLLOWER]: Refusing update from remote peer 295003248359433f92b8afad8501e63d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:56.468710 21692 raft_consensus.cc:1261] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [term 1 FOLLOWER]: Refusing update from remote peer 295003248359433f92b8afad8501e63d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:16:56.469607 21766 consensus_queue.cc:1035] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e170a5bfa8544a3db8cb37167ffbe44a" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:56.469902 21762 consensus_queue.cc:1035] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2731dbbd6145d68e352592c260be29" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 43641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:16:56.480173 21769 mvcc.cc:205] Tried to move back new op lower bound from 7333743069046616064 to 7333743068908904448. Current Snapshot: MvccSnapshot[applied={T|T < 7333743069046616064}]
I20260926 23:16:56.494094 21289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:56.494335 21289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:16:56.497787 21289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ed416e90ce0488c86724c47efb6b874: Sending DeleteTablet for 3 replicas of tablet 86e22f4a10ba4f0a8aedcc78a7d944ef
I20260926 23:16:56.498754 21411 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:56 UTC) from {username='slave'} at 127.0.0.1:35536
I20260926 23:16:56.498864 21672 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:56 UTC) from {username='slave'} at 127.0.0.1:58120
I20260926 23:16:56.499074 21542 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:16:56 UTC) from {username='slave'} at 127.0.0.1:37526
I20260926 23:16:56.501369 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 21349
I20260926 23:16:56.513899 21780 tablet_replica.cc:330] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: stopping tablet replica
I20260926 23:16:56.514060 21779 tablet_replica.cc:330] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: stopping tablet replica
I20260926 23:16:56.514353 21780 raft_consensus.cc:2229] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:56.514433 21779 raft_consensus.cc:2229] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:16:56.514875 21779 raft_consensus.cc:2258] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:56.514864 21780 raft_consensus.cc:2258] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:56.516302 21778 tablet_replica.cc:330] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: stopping tablet replica
I20260926 23:16:56.516671 21778 raft_consensus.cc:2229] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:16:56.517328 21778 raft_consensus.cc:2258] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:16:56.518546 21780 ts_tablet_manager.cc:1905] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:56.519196 21779 ts_tablet_manager.cc:1905] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:56.519642 21778 ts_tablet_manager.cc:1905] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:16:56.524446 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.504128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.16.62.65:46023)
0926 23:16:56.504348 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.504357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.515676 (+ 11319us) server_negotiation.cc:407] Connection header received
0926 23:16:56.515882 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.515890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.515973 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.516088 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.523560 (+ 7472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.523565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.523569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.523588 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.523626 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.523645 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.523745 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.524042 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.524099 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:16:56.525456 21471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:56.525861 21779 ts_tablet_manager.cc:1918] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:16:56.526095 21779 log.cc:1196] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/86e22f4a10ba4f0a8aedcc78a7d944ef
I20260926 23:16:56.526621 21779 ts_tablet_manager.cc:1939] T 86e22f4a10ba4f0a8aedcc78a7d944ef P bf2731dbbd6145d68e352592c260be29: Deleting consensus metadata
I20260926 23:16:56.528355 21275 catalog_manager.cc:4841] TS bf2731dbbd6145d68e352592c260be29 (127.16.62.67:43641): tablet 86e22f4a10ba4f0a8aedcc78a7d944ef (table test-table [id=932b08c2077548bda12ad2ed89d080a4]) successfully deleted
I20260926 23:16:56.528687 21780 ts_tablet_manager.cc:1918] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:16:56.528973 21780 log.cc:1196] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/86e22f4a10ba4f0a8aedcc78a7d944ef
I20260926 23:16:56.529850 21780 ts_tablet_manager.cc:1939] T 86e22f4a10ba4f0a8aedcc78a7d944ef P e170a5bfa8544a3db8cb37167ffbe44a: Deleting consensus metadata
I20260926 23:16:56.531651 21276 catalog_manager.cc:4841] TS e170a5bfa8544a3db8cb37167ffbe44a (127.16.62.66:36917): tablet 86e22f4a10ba4f0a8aedcc78a7d944ef (table test-table [id=932b08c2077548bda12ad2ed89d080a4]) successfully deleted
I20260926 23:16:56.619513 21778 ts_tablet_manager.cc:1918] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:16:56.619784 21778 log.cc:1196] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/86e22f4a10ba4f0a8aedcc78a7d944ef
I20260926 23:16:56.620270 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 21349
I20260926 23:16:56.620486 21778 ts_tablet_manager.cc:1939] T 86e22f4a10ba4f0a8aedcc78a7d944ef P 295003248359433f92b8afad8501e63d: Deleting consensus metadata
W20260926 23:16:56.637284 21275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.62.65:46023 (error 108)
I20260926 23:16:56.638434 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 21480
I20260926 23:16:56.639245 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.638619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:46023 (local address 127.0.0.1:35566)
0926 23:16:56.638827 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.638999 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 23:16:56.639456 21275 catalog_manager.cc:4584] TS 295003248359433f92b8afad8501e63d (127.16.62.65:46023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86e22f4a10ba4f0a8aedcc78a7d944ef: Network error: Client connection negotiation failed: client connection to 127.16.62.65:46023: connect: Connection refused (error 111)
I20260926 23:16:56.641290 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.639239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.16.62.66:36917)
0926 23:16:56.639431 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.639436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.639457 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:16:56.639635 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.639642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.639744 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.639828 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.640756 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.640761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.640764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.640779 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.640808 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.640820 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.640931 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.641036 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.641091 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:16:56.641907 21602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:56.693249 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.692786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:46023 (local address 127.0.0.1:35582)
0926 23:16:56.692999 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.693109 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:16:56.694248 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.693918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:46023 (local address 127.0.0.1:35594)
0926 23:16:56.694043 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.694107 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 23:16:56.727900 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 21480
I20260926 23:16:56.746272 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 21610
I20260926 23:16:56.748883 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.746978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.16.62.67:43641)
0926 23:16:56.747143 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:56.747147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:56.747224 (+ 77us) server_negotiation.cc:407] Connection header received
0926 23:16:56.747373 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:56.747377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:56.747453 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:56.747533 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:56.748331 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:56.748336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:56.748339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:56.748353 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:56.748383 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:56.748394 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:56.748492 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:56.748617 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 23:16:56.748659 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:16:56.749528 21732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:56.754954 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.754584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:46023 (local address 127.0.0.1:35600)
0926 23:16:56.754752 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.754832 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 23:16:56.755883 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.755530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:46023 (local address 127.0.0.1:35606)
0926 23:16:56.755659 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.755754 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 23:16:56.835096 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 21610
I20260926 23:16:56.836884 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.836457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:46023 (local address 127.0.0.1:35612)
0926 23:16:56.836666 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.836749 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 23:16:56.838193 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:56.837539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:46023 (local address 127.0.0.1:35620)
0926 23:16:56.837748 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:56.838013 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 23:16:56.852188 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 21259
I20260926 23:16:56.853256 21320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:56.957103 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 21259
2026-09-26T23:16:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2503 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 23:16:56.997872 16633 mini_oidc.cc:152] Starting JWKS server
I20260926 23:16:57.000578 16633 webserver.cc:460] Webserver started at https://127.0.0.1:43775/ using document root <none> and password file <none>
I20260926 23:16:57.000871 16633 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 23:16:57.001498 16633 webserver.cc:460] Webserver started at http://127.0.0.1:39045/ using document root <none> and password file <none>
2026-09-26T23:16:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:57Z Disabled control of system clock
I20260926 23:16:57.027064 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44457
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:44457
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43775/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/testchainca.cert with env {}
W20260926 23:16:57.333350 21792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:57.333613 21792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:57.333719 21792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:57.340708 21792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:57.340826 21792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:57.340864 21792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:57.340912 21792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:57.340955 21792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:16:57.341086 21792 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:16:57.341156 21792 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43775/jwks.json
I20260926 23:16:57.350915 21792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:44457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43775/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:57.352243 21792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:57.353881 21792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:57.365346 21798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:57.365623 21797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:57.365798 21800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:57.366099 21792 server_base.cc:1034] running on GCE node
I20260926 23:16:57.366858 21792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:57.368300 21792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:57.369518 21792 hybrid_clock.cc:648] HybridClock initialized: now 1790464617369487 us; error 43 us; skew 500 ppm
I20260926 23:16:57.369858 21792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:16:57.371985 21792 webserver.cc:460] Webserver started at http://127.16.62.126:33543/ using document root <none> and password file <none>
I20260926 23:16:57.372725 21792 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:57.372859 21792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:57.373164 21792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:57.375909 21792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "24595108a1d34ef0829f9382aa982826"
format_stamp: "Formatted at 2026-09-26 23:16:57 on dist-test-slave-vcv1"
server_key: "3b47d27a166847a47b5ca80dfc1a603f"
server_key_iv: "7556c647ff3c91e9148bd998d474e743"
server_key_version: "encryptionkey@0"
I20260926 23:16:57.376737 21792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "24595108a1d34ef0829f9382aa982826"
format_stamp: "Formatted at 2026-09-26 23:16:57 on dist-test-slave-vcv1"
server_key: "3b47d27a166847a47b5ca80dfc1a603f"
server_key_iv: "7556c647ff3c91e9148bd998d474e743"
server_key_version: "encryptionkey@0"
I20260926 23:16:57.381359 21792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:16:57.384474 21806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:57.385653 21792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 23:16:57.385833 21792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "24595108a1d34ef0829f9382aa982826"
format_stamp: "Formatted at 2026-09-26 23:16:57 on dist-test-slave-vcv1"
server_key: "3b47d27a166847a47b5ca80dfc1a603f"
server_key_iv: "7556c647ff3c91e9148bd998d474e743"
server_key_version: "encryptionkey@0"
I20260926 23:16:57.386029 21792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:57.412844 21792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:57.413633 21792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:57.413946 21792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:57.431854 21792 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:44457
I20260926 23:16:57.431874 21857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:44457 every 8 connection(s)
I20260926 23:16:57.433157 21792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:57.436637 21858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:57.442839 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 21792
I20260926 23:16:57.443033 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:57.443637 16633 external_mini_cluster.cc:1428] Setting key 116df8503c426d8e51768227d6304a15
I20260926 23:16:57.446945 21858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826: Bootstrap starting.
I20260926 23:16:57.449936 21858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:57.450930 21858 log.cc:824] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:57.453608 21858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826: No bootstrap required, opened a new log
I20260926 23:16:57.456532 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:57.445458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.16.62.126:44457)
0926 23:16:57.446356 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:57.446367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:57.446406 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:16:57.448102 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:57.448123 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:57.448626 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:57.449079 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:57.449621 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:57.450928 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:57.452290 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:57.452928 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:57.453127 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:57.454231 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:57.454238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:57.454250 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:57.454742 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:57.454831 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:57.454850 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:57.454989 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:57.455224 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 23:16:57.455507 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":312,"threads_started":1}
I20260926 23:16:57.460027 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:16:57.461423 21858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24595108a1d34ef0829f9382aa982826" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44457 } }
I20260926 23:16:57.461756 21858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:57.461866 21858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24595108a1d34ef0829f9382aa982826, State: Initialized, Role: FOLLOWER
I20260926 23:16:57.462505 21858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24595108a1d34ef0829f9382aa982826" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44457 } }
I20260926 23:16:57.462725 21858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:57.462888 21858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:57.463060 21858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:57.465328 21858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24595108a1d34ef0829f9382aa982826" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44457 } }
I20260926 23:16:57.465798 21858 leader_election.cc:302] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24595108a1d34ef0829f9382aa982826; no voters:
I20260926 23:16:57.466327 21858 leader_election.cc:288] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:57.466456 21863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:57.467177 21863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [term 1 LEADER]: Becoming Leader. State: Replica: 24595108a1d34ef0829f9382aa982826, State: Running, Role: LEADER
I20260926 23:16:57.467634 21863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24595108a1d34ef0829f9382aa982826" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44457 } }
I20260926 23:16:57.468282 21858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:57.470517 21864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24595108a1d34ef0829f9382aa982826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24595108a1d34ef0829f9382aa982826" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44457 } } }
I20260926 23:16:57.471089 21864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:57.470854 21865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24595108a1d34ef0829f9382aa982826. Latest consensus state: current_term: 1 leader_uuid: "24595108a1d34ef0829f9382aa982826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24595108a1d34ef0829f9382aa982826" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44457 } } }
I20260926 23:16:57.471910 21865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826 [sys.catalog]: This master's current role is: LEADER
W20260926 23:16:57.479656 21879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:16:57.479754 21879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:16:57.479924 21869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:57.482038 21869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:57.490294 21869 catalog_manager.cc:1347] Generated new cluster ID: f894fbb4fd6b42c38cc80e02cef552a9
I20260926 23:16:57.490445 21869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:57.509784 21869 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:57.511016 21869 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:57.518018 21869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24595108a1d34ef0829f9382aa982826: Generated new TSK 0
I20260926 23:16:57.518818 21869 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 23:16:57.561986 21888 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 23:16:57.562191 21888 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 23:16:57.563253 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:57.541271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.16.62.126:44457)
0926 23:16:57.541467 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:57.541474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:57.542039 (+ 565us) server_negotiation.cc:407] Connection header received
0926 23:16:57.542348 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:57.542354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:57.542435 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:57.542583 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:16:57.543091 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:57.544253 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:57.545597 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:57.546306 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:57.546436 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:57.546950 (+ 514us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:16:57.562603 (+ 15653us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 23:16:57.562963 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37380: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 23:16:57.563385 21861 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37380: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 23:16:57.564774 21887 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:16:57.541263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44457 (local address 127.0.0.1:37380)
0926 23:16:57.541935 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:57.541951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:57.542092 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:57.542615 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:57.542618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:57.542628 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:16:57.542944 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:57.542951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:57.544370 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:57.544374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:57.545466 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:57.545477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:57.546472 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:57.546477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:57.546606 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:57.546658 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:16:57.562881 (+ 16223us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 23:16:57.563122 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.62.126:44457: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":352,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.62.126:44457: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 23:16:57.566331 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 21792
I20260926 23:16:57.567462 21853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:57.659832 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 21792
2026-09-26T23:16:57Z chronyd exiting
I20260926 23:16:57.709564 16633 test_util.cc:175] -----------------------------------------------
I20260926 23:16:57.709709 16633 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464575313384-16633-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (716 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 23:16:57.713512 16633 mini_oidc.cc:152] Starting JWKS server
I20260926 23:16:57.715360 16633 webserver.cc:460] Webserver started at https://127.0.0.1:34999/ using document root <none> and password file <none>
I20260926 23:16:57.715612 16633 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 23:16:57.716293 16633 webserver.cc:460] Webserver started at http://127.0.0.1:35641/ using document root <none> and password file <none>
2026-09-26T23:16:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:57Z Disabled control of system clock
I20260926 23:16:57.742053 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:45665
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:45665
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34999/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/testchainca.cert with env {}
W20260926 23:16:58.049899 21895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:58.050158 21895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:58.050239 21895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:58.057148 21895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:58.057235 21895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:58.057271 21895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:58.057324 21895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:58.057365 21895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:16:58.057416 21895 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:16:58.057488 21895 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34999/jwks.json
I20260926 23:16:58.066790 21895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:45665
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:45665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34999/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:58.067998 21895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:58.069223 21895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:58.080493 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:58.080580 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:58.080767 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:58.081568 21895 server_base.cc:1034] running on GCE node
I20260926 23:16:58.082333 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:58.083650 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:58.084868 21895 hybrid_clock.cc:648] HybridClock initialized: now 1790464618084827 us; error 54 us; skew 500 ppm
I20260926 23:16:58.085186 21895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:16:58.087606 21895 webserver.cc:460] Webserver started at http://127.16.62.126:38983/ using document root <none> and password file <none>
I20260926 23:16:58.088485 21895 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:58.088605 21895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:58.088990 21895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:58.093154 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "96336d2c1d9e4840a538c33b86afa849"
format_stamp: "Formatted at 2026-09-26 23:16:58 on dist-test-slave-vcv1"
server_key: "04d96127ab203e0e60d6ab2c11c5b6dc"
server_key_iv: "3f8d0385dd8c148eb8da2fc9f1b42dbb"
server_key_version: "encryptionkey@0"
I20260926 23:16:58.094158 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "96336d2c1d9e4840a538c33b86afa849"
format_stamp: "Formatted at 2026-09-26 23:16:58 on dist-test-slave-vcv1"
server_key: "04d96127ab203e0e60d6ab2c11c5b6dc"
server_key_iv: "3f8d0385dd8c148eb8da2fc9f1b42dbb"
server_key_version: "encryptionkey@0"
I20260926 23:16:58.099669 21895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 23:16:58.102782 21909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:58.104034 21895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 23:16:58.104249 21895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "96336d2c1d9e4840a538c33b86afa849"
format_stamp: "Formatted at 2026-09-26 23:16:58 on dist-test-slave-vcv1"
server_key: "04d96127ab203e0e60d6ab2c11c5b6dc"
server_key_iv: "3f8d0385dd8c148eb8da2fc9f1b42dbb"
server_key_version: "encryptionkey@0"
I20260926 23:16:58.104449 21895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:58.122119 21895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:58.122823 21895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:58.123122 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:58.141058 21895 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:45665
I20260926 23:16:58.141155 21960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:45665 every 8 connection(s)
I20260926 23:16:58.142323 21895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:58.145787 21961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:58.149226 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 21895
I20260926 23:16:58.149479 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:58.150365 16633 external_mini_cluster.cc:1428] Setting key 2ef34b0d810a14244afc81063bef9cf6
I20260926 23:16:58.158690 21961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849: Bootstrap starting.
I20260926 23:16:58.162568 21961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:58.163550 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:58.152266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.16.62.126:45665)
0926 23:16:58.153191 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:58.153201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:58.153249 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:16:58.154780 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:58.154800 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:58.155238 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:58.155527 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:58.156205 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.157429 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.159272 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.159881 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.160018 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:58.161362 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:58.161369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:58.161380 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:58.161880 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:58.161966 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:58.161984 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:58.162066 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:58.162308 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 23:16:58.162589 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":369,"threads_started":1}
I20260926 23:16:58.163719 21961 log.cc:824] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:58.166723 21961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849: No bootstrap required, opened a new log
I20260926 23:16:58.167230 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:16:58.174404 21961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96336d2c1d9e4840a538c33b86afa849" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45665 } }
I20260926 23:16:58.174753 21961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:58.174875 21961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96336d2c1d9e4840a538c33b86afa849, State: Initialized, Role: FOLLOWER
I20260926 23:16:58.175472 21961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96336d2c1d9e4840a538c33b86afa849" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45665 } }
I20260926 23:16:58.175719 21961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:58.175904 21961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:58.176093 21961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:58.178596 21961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96336d2c1d9e4840a538c33b86afa849" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45665 } }
I20260926 23:16:58.179092 21961 leader_election.cc:302] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96336d2c1d9e4840a538c33b86afa849; no voters:
I20260926 23:16:58.179644 21961 leader_election.cc:288] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:58.179744 21966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:58.180538 21966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [term 1 LEADER]: Becoming Leader. State: Replica: 96336d2c1d9e4840a538c33b86afa849, State: Running, Role: LEADER
I20260926 23:16:58.181041 21966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96336d2c1d9e4840a538c33b86afa849" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45665 } }
I20260926 23:16:58.181740 21961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:58.183734 21968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96336d2c1d9e4840a538c33b86afa849. Latest consensus state: current_term: 1 leader_uuid: "96336d2c1d9e4840a538c33b86afa849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96336d2c1d9e4840a538c33b86afa849" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45665 } } }
I20260926 23:16:58.184237 21968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:58.184599 21967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96336d2c1d9e4840a538c33b86afa849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96336d2c1d9e4840a538c33b86afa849" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45665 } } }
I20260926 23:16:58.184939 21967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:58.186373 21972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:58.192968 21972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:58.202754 21972 catalog_manager.cc:1347] Generated new cluster ID: a6e952d87428485ebfe1f0a87d7dff2b
I20260926 23:16:58.202890 21972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:58.220438 21972 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:58.221884 21972 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:58.230348 21972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96336d2c1d9e4840a538c33b86afa849: Generated new TSK 0
I20260926 23:16:58.231110 21972 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 23:16:58.262324 21991 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 23:16:58.262441 21991 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260926 23:16:58.263664 21990 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:16:58.243564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:45665 (local address 127.0.0.1:52482)
0926 23:16:58.244126 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:58.244189 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:58.244311 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:58.244750 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:58.244753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:58.244761 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:16:58.245028 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:58.245035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.246521 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.246525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:58.247579 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:58.247589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.248757 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.248761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:58.248868 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:58.248896 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:16:58.263223 (+ 14327us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 23:16:58.263390 (+ 167us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.62.126:45665: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":309,"threads_started":1}
I20260926 23:16:58.263718 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:58.243706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.16.62.126:45665)
0926 23:16:58.243883 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:58.243890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:58.244281 (+ 391us) server_negotiation.cc:407] Connection header received
0926 23:16:58.244466 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:58.244472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:58.244562 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:58.244748 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:16:58.245184 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.246396 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.247723 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.248607 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.248738 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:58.249167 (+ 429us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:16:58.262995 (+ 13828us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 23:16:58.263418 (+ 423us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 23:16:58.263825 21964 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 23:16:58.264423 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 21895
I20260926 23:16:58.265589 21956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:58.355635 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 21895
2026-09-26T23:16:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (699 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 23:16:58.412559 16633 mini_oidc.cc:152] Starting JWKS server
I20260926 23:16:58.414362 16633 webserver.cc:460] Webserver started at https://127.0.0.1:37833/ using document root <none> and password file <none>
I20260926 23:16:58.414585 16633 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 23:16:58.415189 16633 webserver.cc:460] Webserver started at http://127.0.0.1:35161/ using document root <none> and password file <none>
2026-09-26T23:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:58Z Disabled control of system clock
I20260926 23:16:58.440256 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:38929
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:38543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:38929
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37833/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 23:16:58.749711 21998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:58.749997 21998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:58.750108 21998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:58.757246 21998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:58.757340 21998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:58.757376 21998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:58.757429 21998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:58.757471 21998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:16:58.757529 21998 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:16:58.757566 21998 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37833/jwks.json
I20260926 23:16:58.766194 21998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:38929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:38929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37833/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:58.767333 21998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:58.769013 21998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:58.780419 22004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:58.780376 22003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:58.780468 22006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:58.781363 21998 server_base.cc:1034] running on GCE node
I20260926 23:16:58.782115 21998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:58.783434 21998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:58.784651 21998 hybrid_clock.cc:648] HybridClock initialized: now 1790464618784614 us; error 25 us; skew 500 ppm
I20260926 23:16:58.784988 21998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:16:58.799247 21998 webserver.cc:460] Webserver started at http://127.16.62.126:43651/ using document root <none> and password file <none>
I20260926 23:16:58.799968 21998 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:58.800107 21998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:58.800536 21998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:58.803385 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "33716805c5cf4d22bbd1ef0c0b559b8e"
format_stamp: "Formatted at 2026-09-26 23:16:58 on dist-test-slave-vcv1"
server_key: "bfb8f369d4db35df994b7de707ec69ab"
server_key_iv: "61270ad0b543327e8963814140ff7863"
server_key_version: "encryptionkey@0"
I20260926 23:16:58.804234 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "33716805c5cf4d22bbd1ef0c0b559b8e"
format_stamp: "Formatted at 2026-09-26 23:16:58 on dist-test-slave-vcv1"
server_key: "bfb8f369d4db35df994b7de707ec69ab"
server_key_iv: "61270ad0b543327e8963814140ff7863"
server_key_version: "encryptionkey@0"
I20260926 23:16:58.809049 21998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:16:58.812268 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:58.813486 21998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:16:58.813660 21998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "33716805c5cf4d22bbd1ef0c0b559b8e"
format_stamp: "Formatted at 2026-09-26 23:16:58 on dist-test-slave-vcv1"
server_key: "bfb8f369d4db35df994b7de707ec69ab"
server_key_iv: "61270ad0b543327e8963814140ff7863"
server_key_version: "encryptionkey@0"
I20260926 23:16:58.813863 21998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:58.832791 21998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:58.833539 21998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:58.833838 21998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:58.851065 21998 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:38929
I20260926 23:16:58.851096 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:38929 every 8 connection(s)
I20260926 23:16:58.852327 21998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:58.855844 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:58.857125 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 21998
I20260926 23:16:58.857326 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:58.857968 16633 external_mini_cluster.cc:1428] Setting key 9592d943fef11ff5b36157cd2dc64381
I20260926 23:16:58.867869 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e: Bootstrap starting.
I20260926 23:16:58.870399 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:58.859716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.16.62.126:38929)
0926 23:16:58.860733 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:58.860742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:58.860783 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:16:58.862304 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:58.862324 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:58.862758 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:58.863113 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:58.863667 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.864963 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.866252 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.866862 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.867067 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:58.868278 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:58.868286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:58.868297 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:58.868792 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:58.868865 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:58.868886 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:58.869006 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:58.869240 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 23:16:58.869514 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":410,"threads_started":1}
I20260926 23:16:58.871173 22064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:58.872285 22064 log.cc:824] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:58.874249 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:16:58.875077 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e: No bootstrap required, opened a new log
I20260926 23:16:58.881901 22064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33716805c5cf4d22bbd1ef0c0b559b8e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38929 } }
I20260926 23:16:58.882259 22064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:58.882362 22064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33716805c5cf4d22bbd1ef0c0b559b8e, State: Initialized, Role: FOLLOWER
I20260926 23:16:58.882874 22064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33716805c5cf4d22bbd1ef0c0b559b8e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38929 } }
I20260926 23:16:58.883101 22064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:58.883250 22064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:58.883414 22064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:58.885538 22064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33716805c5cf4d22bbd1ef0c0b559b8e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38929 } }
I20260926 23:16:58.885999 22064 leader_election.cc:302] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33716805c5cf4d22bbd1ef0c0b559b8e; no voters:
I20260926 23:16:58.886551 22064 leader_election.cc:288] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:58.886698 22069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:58.887483 22069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [term 1 LEADER]: Becoming Leader. State: Replica: 33716805c5cf4d22bbd1ef0c0b559b8e, State: Running, Role: LEADER
I20260926 23:16:58.887993 22069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33716805c5cf4d22bbd1ef0c0b559b8e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38929 } }
I20260926 23:16:58.888612 22064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:58.890972 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33716805c5cf4d22bbd1ef0c0b559b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33716805c5cf4d22bbd1ef0c0b559b8e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38929 } } }
I20260926 23:16:58.891422 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:58.891942 22071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33716805c5cf4d22bbd1ef0c0b559b8e. Latest consensus state: current_term: 1 leader_uuid: "33716805c5cf4d22bbd1ef0c0b559b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33716805c5cf4d22bbd1ef0c0b559b8e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38929 } } }
I20260926 23:16:58.896293 22071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:58.897665 22076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:58.903126 22076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:58.911198 22076 catalog_manager.cc:1347] Generated new cluster ID: 00d4f0c9fd5b4cdba37651fff9881874
I20260926 23:16:58.911334 22076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:58.922808 22076 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:58.923924 22076 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:58.930123 22076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33716805c5cf4d22bbd1ef0c0b559b8e: Generated new TSK 0
I20260926 23:16:58.930810 22076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:16:58.956301 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:58.948118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.16.62.126:38929)
0926 23:16:58.948301 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:58.948308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:58.948732 (+ 424us) server_negotiation.cc:407] Connection header received
0926 23:16:58.948955 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:58.948960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:58.949048 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:58.949202 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:16:58.949589 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.950706 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.952040 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.952744 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.952879 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:58.953354 (+ 475us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:16:58.955644 (+ 2290us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 23:16:58.955980 (+ 336us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60394: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 23:16:58.956434 22067 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60394: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 23:16:58.958113 22093 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:16:58.948024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38929 (local address 127.0.0.1:60394)
0926 23:16:58.948640 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:58.948654 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:58.948772 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:58.949205 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:58.949208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:58.949217 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:16:58.949460 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:58.949467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.950857 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.950860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:58.951908 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:58.951918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:58.952918 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:58.952934 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:58.953059 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:58.953088 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:16:58.955856 (+ 2768us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 23:16:58.957679 (+ 1823us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.62.126:38929: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":1542,"thread_start_us":356,"threads_started":1}
I20260926 23:16:58.959425 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 21998
I20260926 23:16:58.960623 22059 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 23:16:58.968473 22094 webserver.cc:533] Webserver: sslize failed:
E20260926 23:16:58.968552 22094 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 23:16:59.049319 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 21998
2026-09-26T23:16:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (691 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T23:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:16:59Z Disabled control of system clock
I20260926 23:16:59.129490 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:43667
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:43667
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/testchainca.cert with env {}
W20260926 23:16:59.446074 22099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:16:59.446347 22099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:16:59.446430 22099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:16:59.453498 22099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:16:59.453591 22099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:16:59.453627 22099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:16:59.453680 22099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:16:59.453722 22099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:16:59.453771 22099 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:16:59.453809 22099 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 23:16:59.463279 22099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:43667
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:43667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:16:59.464453 22099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:16:59.465626 22099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:16:59.476764 22104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:59.476790 22107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:16:59.476778 22105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:16:59.477890 22099 server_base.cc:1034] running on GCE node
I20260926 23:16:59.478641 22099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:16:59.479857 22099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:16:59.481053 22099 hybrid_clock.cc:648] HybridClock initialized: now 1790464619480990 us; error 59 us; skew 500 ppm
I20260926 23:16:59.481392 22099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:16:59.483557 22099 webserver.cc:460] Webserver started at http://127.16.62.126:37757/ using document root <none> and password file <none>
I20260926 23:16:59.484266 22099 fs_manager.cc:359] Metadata directory not provided
I20260926 23:16:59.484375 22099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:16:59.484678 22099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:16:59.487416 22099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "7afb3eec50614b3f907eac375f24b425"
format_stamp: "Formatted at 2026-09-26 23:16:59 on dist-test-slave-vcv1"
server_key: "5a29e34005f99cfe3ddd19f7137c427b"
server_key_iv: "5382ba98228e5266d0dbef48bd84965b"
server_key_version: "encryptionkey@0"
I20260926 23:16:59.488277 22099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "7afb3eec50614b3f907eac375f24b425"
format_stamp: "Formatted at 2026-09-26 23:16:59 on dist-test-slave-vcv1"
server_key: "5a29e34005f99cfe3ddd19f7137c427b"
server_key_iv: "5382ba98228e5266d0dbef48bd84965b"
server_key_version: "encryptionkey@0"
I20260926 23:16:59.493057 22099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:16:59.496351 22113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:16:59.497670 22099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:16:59.497859 22099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "7afb3eec50614b3f907eac375f24b425"
format_stamp: "Formatted at 2026-09-26 23:16:59 on dist-test-slave-vcv1"
server_key: "5a29e34005f99cfe3ddd19f7137c427b"
server_key_iv: "5382ba98228e5266d0dbef48bd84965b"
server_key_version: "encryptionkey@0"
I20260926 23:16:59.498097 22099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:16:59.513230 22099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:16:59.514060 22099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:16:59.514356 22099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:16:59.532642 22099 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:43667
I20260926 23:16:59.532662 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:43667 every 8 connection(s)
I20260926 23:16:59.534000 22099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:16:59.537599 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:16:59.538565 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 22099
I20260926 23:16:59.538796 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:16:59.539579 16633 external_mini_cluster.cc:1428] Setting key 7003c96a2fd3b6d417f733dd39566851
I20260926 23:16:59.549857 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425: Bootstrap starting.
I20260926 23:16:59.552251 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:59.541493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.16.62.126:43667)
0926 23:16:59.542428 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:59.542438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:59.542482 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:16:59.544057 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:59.544078 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:59.544594 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:59.544986 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:16:59.545499 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:59.546687 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:59.547949 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:59.548641 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:59.548850 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:59.550045 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:16:59.550052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:16:59.550066 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:16:59.550563 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:16:59.550632 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:16:59.550651 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:16:59.550793 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 23:16:59.551007 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 23:16:59.551301 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":324,"threads_started":1}
I20260926 23:16:59.552999 22165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425: Neither blocks nor log segments found. Creating new log.
I20260926 23:16:59.553979 22165 log.cc:824] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425: Log is configured to *not* fsync() on all Append() calls
I20260926 23:16:59.555789 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:16:59.556842 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425: No bootstrap required, opened a new log
I20260926 23:16:59.564580 22165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afb3eec50614b3f907eac375f24b425" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43667 } }
I20260926 23:16:59.564949 22165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:16:59.565070 22165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7afb3eec50614b3f907eac375f24b425, State: Initialized, Role: FOLLOWER
I20260926 23:16:59.565594 22165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afb3eec50614b3f907eac375f24b425" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43667 } }
I20260926 23:16:59.565822 22165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:16:59.565959 22165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:16:59.566129 22165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:16:59.568243 22165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afb3eec50614b3f907eac375f24b425" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43667 } }
I20260926 23:16:59.568717 22165 leader_election.cc:302] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7afb3eec50614b3f907eac375f24b425; no voters:
I20260926 23:16:59.569367 22165 leader_election.cc:288] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:16:59.569562 22170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:16:59.570628 22170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [term 1 LEADER]: Becoming Leader. State: Replica: 7afb3eec50614b3f907eac375f24b425, State: Running, Role: LEADER
I20260926 23:16:59.571107 22170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afb3eec50614b3f907eac375f24b425" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43667 } }
I20260926 23:16:59.571786 22165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:16:59.573807 22172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7afb3eec50614b3f907eac375f24b425. Latest consensus state: current_term: 1 leader_uuid: "7afb3eec50614b3f907eac375f24b425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afb3eec50614b3f907eac375f24b425" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43667 } } }
I20260926 23:16:59.573760 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7afb3eec50614b3f907eac375f24b425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afb3eec50614b3f907eac375f24b425" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 43667 } } }
I20260926 23:16:59.574126 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:59.574126 22172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425 [sys.catalog]: This master's current role is: LEADER
I20260926 23:16:59.576399 22177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:16:59.580400 22177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:16:59.588572 22177 catalog_manager.cc:1347] Generated new cluster ID: 0c6abc30fd9e48499d74653fb294e86c
I20260926 23:16:59.588675 22177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:16:59.606572 22177 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:16:59.607735 22177 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:16:59.619484 22177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7afb3eec50614b3f907eac375f24b425: Generated new TSK 0
I20260926 23:16:59.620383 22177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:16:59.689225 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:16:59.640935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.16.62.126:43667)
0926 23:16:59.641112 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:16:59.641119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:16:59.641484 (+ 365us) server_negotiation.cc:407] Connection header received
0926 23:16:59.641730 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:16:59.641736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:16:59.641829 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:16:59.641992 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:16:59.642402 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:59.643499 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:59.644876 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:16:59.645542 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:16:59.645665 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:59.646105 (+ 440us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:16:59.688496 (+ 42391us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 23:16:59.688910 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 23:16:59.689375 22168 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 23:16:59.689769 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:16:59.640810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:43667 (local address 127.0.0.1:53866)
0926 23:16:59.641388 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 23:16:59.641403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:16:59.641521 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:16:59.642006 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:16:59.642010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:16:59.642019 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:16:59.642260 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:59.642267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:59.643639 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:59.643642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:59.644747 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:16:59.644758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:16:59.645703 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:16:59.645708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:16:59.645832 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:16:59.645862 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:16:59.688795 (+ 42933us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 23:16:59.689075 (+ 280us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.62.126:43667: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":304,"threads_started":1}
I20260926 23:16:59.696807 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 22099
I20260926 23:16:59.698086 22160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:16:59.798163 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 22099
2026-09-26T23:16:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (728 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (400 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (395 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:17:00 dist-test-slave-vcv1 krb5kdc[22209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:17:00 dist-test-slave-vcv1 krb5kdc[22209](info): set up 2 sockets
Sep 26 23:17:00 dist-test-slave-vcv1 krb5kdc[22209](info): commencing operation
krb5kdc: starting...
W20260926 23:17:01.697571 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:17:01 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:17:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:17:01.893784 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:45609
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:45609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:02.202308 22222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:02.202560 22222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:02.202730 22222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:02.209597 22222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:02.209692 22222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:02.209800 22222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:02.209906 22222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:02.209970 22222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:02.219166 22222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:45609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:45609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:02.220363 22222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:02.221612 22222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:02.233068 22230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:02.233093 22228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:02.233392 22227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:02.233768 22222 server_base.cc:1034] running on GCE node
I20260926 23:17:02.234493 22222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:02.235806 22222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:02.237022 22222 hybrid_clock.cc:648] HybridClock initialized: now 1790464622236979 us; error 42 us; skew 500 ppm
I20260926 23:17:02.237353 22222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:02 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:02.240309 22222 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:17:02.241613 22222 webserver.cc:460] Webserver started at http://127.16.62.126:43549/ using document root <none> and password file <none>
I20260926 23:17:02.242274 22222 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:02.242412 22222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:02.242712 22222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:02.245659 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "b701f3cfd4194d13b5361dbfc5b818ca"
format_stamp: "Formatted at 2026-09-26 23:17:02 on dist-test-slave-vcv1"
server_key: "c94b115e933b99950fab5142571a143e"
server_key_iv: "91d8936ba1fdac80eaad2b6a6a6a3af4"
server_key_version: "encryptionkey@0"
I20260926 23:17:02.246412 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "b701f3cfd4194d13b5361dbfc5b818ca"
format_stamp: "Formatted at 2026-09-26 23:17:02 on dist-test-slave-vcv1"
server_key: "c94b115e933b99950fab5142571a143e"
server_key_iv: "91d8936ba1fdac80eaad2b6a6a6a3af4"
server_key_version: "encryptionkey@0"
I20260926 23:17:02.251269 22222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 23:17:02.254487 22237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:02.255707 22222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:17:02.255892 22222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "b701f3cfd4194d13b5361dbfc5b818ca"
format_stamp: "Formatted at 2026-09-26 23:17:02 on dist-test-slave-vcv1"
server_key: "c94b115e933b99950fab5142571a143e"
server_key_iv: "91d8936ba1fdac80eaad2b6a6a6a3af4"
server_key_version: "encryptionkey@0"
I20260926 23:17:02.256078 22222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:02.278702 22222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:02.282793 22222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:02.283103 22222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:02.302518 22222 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:45609
I20260926 23:17:02.302549 22288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:45609 every 8 connection(s)
I20260926 23:17:02.303885 22222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:17:02.307422 22289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:02.311638 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 22222
I20260926 23:17:02.311893 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:17:02.312569 16633 external_mini_cluster.cc:1428] Setting key e3613b74b911b3bf25817b687d303e14
I20260926 23:17:02.317832 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca: Bootstrap starting.
I20260926 23:17:02.321364 22289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:02.322522 22289 log.cc:824] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:02.325915 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca: No bootstrap required, opened a new log
Sep 26 23:17:02 dist-test-slave-vcv1 krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:02.336720 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:02.314467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.16.62.126:45609)
0926 23:17:02.315269 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:02.315279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:02.315316 (+ 37us) server_negotiation.cc:407] Connection header received
0926 23:17:02.316980 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:02.317002 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:02.317362 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:02.317671 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:02.319423 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:02.320760 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:02.322570 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:02.323543 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:02.323769 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:02.328427 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:02.328441 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:02.328463 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:02.328513 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:02.332446 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:02.333031 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:02.333044 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:02.333051 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:02.333248 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:02.333634 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:02.333639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:02.333643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:02.334316 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:02.334630 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:02.334938 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:17:02.335378 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":391,"threads_started":1}
I20260926 23:17:02.336601 22289 raft_consensus.cc:348] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701f3cfd4194d13b5361dbfc5b818ca" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45609 } }
I20260926 23:17:02.337044 22289 raft_consensus.cc:374] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:02.337193 22289 raft_consensus.cc:729] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b701f3cfd4194d13b5361dbfc5b818ca, State: Initialized, Role: FOLLOWER
I20260926 23:17:02.337719 22289 consensus_queue.cc:260] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701f3cfd4194d13b5361dbfc5b818ca" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45609 } }
I20260926 23:17:02.337960 22289 raft_consensus.cc:388] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:17:02.338120 22289 raft_consensus.cc:482] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:17:02.338303 22289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:02.340561 22289 raft_consensus.cc:504] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701f3cfd4194d13b5361dbfc5b818ca" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45609 } }
I20260926 23:17:02.341006 22289 leader_election.cc:302] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b701f3cfd4194d13b5361dbfc5b818ca; no voters:
I20260926 23:17:02.341534 22289 leader_election.cc:288] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:17:02.341651 22294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:02.342394 22294 raft_consensus.cc:686] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [term 1 LEADER]: Becoming Leader. State: Replica: b701f3cfd4194d13b5361dbfc5b818ca, State: Running, Role: LEADER
I20260926 23:17:02.342952 22294 consensus_queue.cc:237] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701f3cfd4194d13b5361dbfc5b818ca" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45609 } }
I20260926 23:17:02.343487 22289 sys_catalog.cc:564] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:02.345589 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b701f3cfd4194d13b5361dbfc5b818ca. Latest consensus state: current_term: 1 leader_uuid: "b701f3cfd4194d13b5361dbfc5b818ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701f3cfd4194d13b5361dbfc5b818ca" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45609 } } }
I20260926 23:17:02.346002 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:02.347289 22295 sys_catalog.cc:455] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b701f3cfd4194d13b5361dbfc5b818ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701f3cfd4194d13b5361dbfc5b818ca" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 45609 } } }
I20260926 23:17:02.347702 22295 sys_catalog.cc:458] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:02.348083 22300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:17:02.352079 22300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:17:02.360301 22300 catalog_manager.cc:1347] Generated new cluster ID: 39ec60a3e1964f658632b793deda546d
I20260926 23:17:02.360407 22300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:17:02.371527 22300 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:17:02.373252 22300 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:17:02.389500 22300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b701f3cfd4194d13b5361dbfc5b818ca: Generated new TSK 0
I20260926 23:17:02.390373 22300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:17:02.443048 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:45609
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:02.757426 22315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:02.757658 22315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:02.757825 22315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:02.764917 22315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:02.765039 22315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:02.765209 22315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:17:02.780650 22315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:45609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:02.782356 22315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:02.784121 22315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:02.799623 22321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:02.799674 22323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:02.799875 22320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:02.800287 22315 server_base.cc:1034] running on GCE node
I20260926 23:17:02.801096 22315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:02.802251 22315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:02.803489 22315 hybrid_clock.cc:648] HybridClock initialized: now 1790464622803416 us; error 90 us; skew 500 ppm
I20260926 23:17:02.803866 22315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:02 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:02.807593 22315 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:17:02.809058 22315 webserver.cc:460] Webserver started at http://127.16.62.65:42027/ using document root <none> and password file <none>
I20260926 23:17:02.809736 22315 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:02.809864 22315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:02.810177 22315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:02.812959 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "803eed2133ff47f5b4ea524b459a521e"
format_stamp: "Formatted at 2026-09-26 23:17:02 on dist-test-slave-vcv1"
server_key: "1e1596fcefb35503e87fb119d4946360"
server_key_iv: "0933c77b5499101b1bcbb8e9a4e1c781"
server_key_version: "encryptionkey@0"
I20260926 23:17:02.813711 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "803eed2133ff47f5b4ea524b459a521e"
format_stamp: "Formatted at 2026-09-26 23:17:02 on dist-test-slave-vcv1"
server_key: "1e1596fcefb35503e87fb119d4946360"
server_key_iv: "0933c77b5499101b1bcbb8e9a4e1c781"
server_key_version: "encryptionkey@0"
I20260926 23:17:02.818769 22315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:17:02.822490 22330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:02.823861 22315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:17:02.824031 22315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "803eed2133ff47f5b4ea524b459a521e"
format_stamp: "Formatted at 2026-09-26 23:17:02 on dist-test-slave-vcv1"
server_key: "1e1596fcefb35503e87fb119d4946360"
server_key_iv: "0933c77b5499101b1bcbb8e9a4e1c781"
server_key_version: "encryptionkey@0"
I20260926 23:17:02.824281 22315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:02.840554 22315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:02.844612 22315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:02.844878 22315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:02.845724 22315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:02.846951 22315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:02.847050 22315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:02.847198 22315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:02.847268 22315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:02.880681 22315 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:42743
I20260926 23:17:02.880724 22442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:42743 every 8 connection(s)
I20260926 23:17:02.881908 22315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:17:02.890659 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 22315
I20260926 23:17:02.890900 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:17:02.891631 16633 external_mini_cluster.cc:1428] Setting key 343fbcd6c5997f29c2559b33febe494a
Sep 26 23:17:02 dist-test-slave-vcv1 krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:02.903439 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:02.884380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:60079 (local address 127.16.62.126:45609)
0926 23:17:02.884811 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:02.884816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:02.885646 (+ 830us) server_negotiation.cc:407] Connection header received
0926 23:17:02.887362 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:02.887368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:02.887456 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:02.887574 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:02.889077 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:02.889982 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:02.892373 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:02.892931 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:02.893062 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:02.896987 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:02.896996 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:02.897016 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:02.897057 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:02.900369 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:02.901283 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:02.901287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:02.901289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:02.901391 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:02.901857 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:02.901859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:02.901861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:02.902096 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:02.902195 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:02.903038 (+ 843us) server_negotiation.cc:299] Negotiation successful
0926 23:17:02.903196 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":234,"threads_started":1}
I20260926 23:17:02.904934 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:02.884820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:45609 (local address 127.16.62.65:60079)
0926 23:17:02.885492 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:02.885533 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:02.887106 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:02.887805 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:02.887812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:02.888350 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:02.888914 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:02.888931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:02.890144 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:02.890148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:02.892212 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:02.892226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:02.893168 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:02.893175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:02.893405 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:02.894064 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:02.894092 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:02.896763 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:02.900494 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:02.900502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:02.900523 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:02.901154 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:02.901496 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:02.901504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:02.901510 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:02.901750 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:02.902213 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:02.902224 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:02.902669 (+ 445us) client_negotiation.cc:769] Sending connection context
0926 23:17:02.902956 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 23:17:02.903494 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":274,"threads_started":1}
I20260926 23:17:02.906402 22443 heartbeater.cc:347] Connected to a master server at 127.16.62.126:45609
I20260926 23:17:02.906700 22443 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:02.907383 22443 heartbeater.cc:511] Master 127.16.62.126:45609 requested a full tablet report, sending...
I20260926 23:17:02.909303 22254 ts_manager.cc:194] Registered new tserver with Master: 803eed2133ff47f5b4ea524b459a521e (127.16.62.65:42743)
I20260926 23:17:02.910631 22254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:60079
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:17:02.937464 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:45609
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:03.246697 22449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:03.246942 22449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:03.247110 22449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:03.254000 22449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:03.254112 22449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:03.254313 22449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:17:03.263427 22449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:45609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:03.264675 22449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:03.265908 22449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:03.278231 22457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:03.278288 22455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:03.278318 22454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:03.279120 22449 server_base.cc:1034] running on GCE node
I20260926 23:17:03.279680 22449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:03.280591 22449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:03.281800 22449 hybrid_clock.cc:648] HybridClock initialized: now 1790464623281768 us; error 44 us; skew 500 ppm
I20260926 23:17:03.282096 22449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:03.284981 22449 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:17:03.286207 22449 webserver.cc:460] Webserver started at http://127.16.62.66:35201/ using document root <none> and password file <none>
I20260926 23:17:03.286899 22449 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:03.287038 22449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:03.287302 22449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:03.289906 22449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "47d464bd6c294c898a8afe9554e7ed5a"
format_stamp: "Formatted at 2026-09-26 23:17:03 on dist-test-slave-vcv1"
server_key: "c9d4f038c423d2382484c20242b00498"
server_key_iv: "4d7cb38b72e441d048779c4679853896"
server_key_version: "encryptionkey@0"
I20260926 23:17:03.290608 22449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "47d464bd6c294c898a8afe9554e7ed5a"
format_stamp: "Formatted at 2026-09-26 23:17:03 on dist-test-slave-vcv1"
server_key: "c9d4f038c423d2382484c20242b00498"
server_key_iv: "4d7cb38b72e441d048779c4679853896"
server_key_version: "encryptionkey@0"
I20260926 23:17:03.295306 22449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 23:17:03.298390 22464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:03.299582 22449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:17:03.299741 22449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "47d464bd6c294c898a8afe9554e7ed5a"
format_stamp: "Formatted at 2026-09-26 23:17:03 on dist-test-slave-vcv1"
server_key: "c9d4f038c423d2382484c20242b00498"
server_key_iv: "4d7cb38b72e441d048779c4679853896"
server_key_version: "encryptionkey@0"
I20260926 23:17:03.299933 22449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:03.327183 22449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:03.331274 22449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:03.331548 22449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:03.332489 22449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:03.333848 22449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:03.333917 22449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:03.334044 22449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:03.334095 22449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:03.366992 22449 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:38907
I20260926 23:17:03.367100 22576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:38907 every 8 connection(s)
I20260926 23:17:03.368281 22449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:17:03.373858 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 22449
I20260926 23:17:03.374055 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:17:03.374615 16633 external_mini_cluster.cc:1428] Setting key e3feda12ee09f8120eaee828689a2eb2
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:03.390925 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:03.370793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:49593 (local address 127.16.62.126:45609)
0926 23:17:03.370998 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:03.371004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:03.372215 (+ 1211us) server_negotiation.cc:407] Connection header received
0926 23:17:03.374238 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:03.374242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:03.374307 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:03.374391 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:03.376260 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.377087 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.379627 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.380020 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.380112 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:03.384203 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:03.384211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:03.384215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:03.384257 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:03.387745 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.388464 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.388472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.388477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.388640 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.389016 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.389023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.389030 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.389388 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:03.389534 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:03.390257 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 23:17:03.390491 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:17:03.391450 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:03.371379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:45609 (local address 127.16.62.66:49593)
0926 23:17:03.372040 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:03.372087 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:03.373959 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:03.374675 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:03.374687 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:03.375389 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:03.376051 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:03.376069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.377247 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.377255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:03.379460 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:03.379477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.380255 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.380264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:03.380612 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:03.381353 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:03.381386 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:03.383963 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:03.387872 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:03.387878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:03.387889 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:03.388344 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:03.388745 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:03.388748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:03.388750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:03.388901 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:03.389496 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:03.389520 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:03.389911 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 23:17:03.390112 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 23:17:03.390443 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":262,"threads_started":1}
I20260926 23:17:03.392900 22577 heartbeater.cc:347] Connected to a master server at 127.16.62.126:45609
I20260926 23:17:03.393196 22577 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:03.393853 22577 heartbeater.cc:511] Master 127.16.62.126:45609 requested a full tablet report, sending...
I20260926 23:17:03.395215 22254 ts_manager.cc:194] Registered new tserver with Master: 47d464bd6c294c898a8afe9554e7ed5a (127.16.62.66:38907)
I20260926 23:17:03.395973 22254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:49593
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:17:03.420678 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:45609
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:03.730334 22582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:03.730654 22582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:03.730822 22582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:03.737816 22582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:03.737955 22582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:03.738134 22582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:17:03.750411 22582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:45609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:03.751709 22582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:03.753059 22582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:03.766887 22588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:03.766842 22590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:03.767305 22587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:03.767406 22582 server_base.cc:1034] running on GCE node
I20260926 23:17:03.768191 22582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:03.769114 22582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:03.770301 22582 hybrid_clock.cc:648] HybridClock initialized: now 1790464623770266 us; error 29 us; skew 500 ppm
I20260926 23:17:03.770607 22582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:03.773614 22582 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:17:03.774979 22582 webserver.cc:460] Webserver started at http://127.16.62.67:46189/ using document root <none> and password file <none>
I20260926 23:17:03.775645 22582 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:03.775796 22582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:03.776095 22582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:03.778900 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "e677d78d465845ebbe789d3ee0e5999b"
format_stamp: "Formatted at 2026-09-26 23:17:03 on dist-test-slave-vcv1"
server_key: "973a43cb849817655cc2617e79037fa4"
server_key_iv: "cc4786912de02dc014508c1a01abcf34"
server_key_version: "encryptionkey@0"
I20260926 23:17:03.779652 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "e677d78d465845ebbe789d3ee0e5999b"
format_stamp: "Formatted at 2026-09-26 23:17:03 on dist-test-slave-vcv1"
server_key: "973a43cb849817655cc2617e79037fa4"
server_key_iv: "cc4786912de02dc014508c1a01abcf34"
server_key_version: "encryptionkey@0"
I20260926 23:17:03.784504 22582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 23:17:03.787765 22597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:03.789017 22582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:03.789213 22582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "e677d78d465845ebbe789d3ee0e5999b"
format_stamp: "Formatted at 2026-09-26 23:17:03 on dist-test-slave-vcv1"
server_key: "973a43cb849817655cc2617e79037fa4"
server_key_iv: "cc4786912de02dc014508c1a01abcf34"
server_key_version: "encryptionkey@0"
I20260926 23:17:03.789443 22582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:03.817855 22582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:03.821636 22582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:03.821921 22582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:03.822829 22582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:03.824064 22582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:03.824214 22582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:03.824359 22582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:03.824429 22582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:03.856765 22582 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:41113
I20260926 23:17:03.856843 22709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:41113 every 8 connection(s)
I20260926 23:17:03.857940 22582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:17:03.867877 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 22582
I20260926 23:17:03.868059 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:17:03.868623 16633 external_mini_cluster.cc:1428] Setting key bd1069e1aeb23d4f76e84b545329558e
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:03.876773 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:03.860343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:34265 (local address 127.16.62.126:45609)
0926 23:17:03.860504 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:03.860508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:03.861610 (+ 1102us) server_negotiation.cc:407] Connection header received
0926 23:17:03.863313 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:03.863317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:03.863383 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:03.863463 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:03.864873 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.865599 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.867065 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.867455 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.867558 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:03.871399 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:03.871403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:03.871406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:03.871456 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:03.874085 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.874790 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.874793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.874795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.874894 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.875238 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.875241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.875242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.875466 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:03.875595 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:03.876373 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 23:17:03.876527 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 23:17:03.877653 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:03.860806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:45609 (local address 127.16.62.67:34265)
0926 23:17:03.861455 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:03.861490 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:03.863051 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:03.863646 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:03.863671 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:03.864124 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:03.864717 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:03.864732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.865761 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.865765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:03.866939 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:03.866950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.867690 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.867700 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:03.867995 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:03.868868 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:03.868896 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:03.871209 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:03.874211 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:03.874218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:03.874228 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:03.874676 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:03.874977 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:03.874980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:03.874982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:03.875148 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:03.875625 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:03.875631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:03.875966 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 23:17:03.876271 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 23:17:03.876622 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":272,"threads_started":1}
I20260926 23:17:03.878942 22710 heartbeater.cc:347] Connected to a master server at 127.16.62.126:45609
I20260926 23:17:03.879266 22710 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:03.880062 22710 heartbeater.cc:511] Master 127.16.62.126:45609 requested a full tablet report, sending...
I20260926 23:17:03.881572 22254 ts_manager.cc:194] Registered new tserver with Master: e677d78d465845ebbe789d3ee0e5999b (127.16.62.67:41113)
I20260926 23:17:03.882679 22254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:34265
I20260926 23:17:03.883975 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/test-admin.keytab.
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:03.909571 16633 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260926 23:17:03.913457 22443 heartbeater.cc:503] Master 127.16.62.126:45609 was elected leader, sending a full tablet report...
W20260926 23:17:03.914933 22717 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:03.912271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.16.62.65:42743)
0926 23:17:03.912929 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:03.912941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:03.912977 (+ 36us) server_negotiation.cc:407] Connection header received
0926 23:17:03.913061 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:03.913069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:03.913282 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:03.913487 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:03.914542 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48030: BlockingRecv error: recv got EOF from 127.0.0.1:48030 (error 108)
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":319,"threads_started":1}
W20260926 23:17:03.915105 16633 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5cc' of type FILE
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:03.916826 16633 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:03.918257 16633 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:17:03.933995 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:03.919788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.16.62.65:42743)
0926 23:17:03.919990 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:03.919997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:03.920292 (+ 295us) server_negotiation.cc:407] Connection header received
0926 23:17:03.920481 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:03.920487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:03.920555 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:03.920665 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:03.921684 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.922672 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.923970 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.924648 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.924797 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:03.928463 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:03.928473 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:03.928486 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:03.928516 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:03.931170 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.931836 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.931849 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.931851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.931963 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.932422 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.932429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.932433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.932786 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:03.933105 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:03.933475 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 23:17:03.933722 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 23:17:03.939677 22717 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:03.937889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.16.62.65:42743)
0926 23:17:03.938109 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:03.938116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:03.938349 (+ 233us) server_negotiation.cc:407] Connection header received
0926 23:17:03.938552 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:03.938558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:03.938634 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:03.938750 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:03.939419 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48048: BlockingRecv error: recv got EOF from 127.0.0.1:48048 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 23:17:03.939888 16633 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5cc' of type FILE
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:03.941787 16633 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:03.943274 16633 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:17:03.956594 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:03.944537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.16.62.65:42743)
0926 23:17:03.944764 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:03.944769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:03.944990 (+ 221us) server_negotiation.cc:407] Connection header received
0926 23:17:03.945087 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:03.945090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:03.945134 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:03.945199 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:03.946059 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.946679 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.947884 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.948352 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.948481 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:03.951769 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:03.951774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:03.951777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:03.951814 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:03.954248 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.954865 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.954872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.954881 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.954983 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.955354 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.955361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.955365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.955603 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:03.955880 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:03.956176 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 23:17:03.956367 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260926 23:17:03.960515 16633 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464575313384-16633-0/krb5kdc/krb5cc' of type FILE
W20260926 23:17:03.961244 22717 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:03.958921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.16.62.65:42743)
0926 23:17:03.959093 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:03.959097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:03.959213 (+ 116us) server_negotiation.cc:407] Connection header received
0926 23:17:03.959336 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:03.959339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:03.959396 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:03.959469 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:03.960096 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48072: BlockingRecv error: recv got EOF from 127.0.0.1:48072 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:03.961966 16633 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:03.963184 16633 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 23:17:03 dist-test-slave-vcv1 krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:17:03.976330 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:03.964419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.16.62.65:42743)
0926 23:17:03.964655 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:03.964662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:03.964822 (+ 160us) server_negotiation.cc:407] Connection header received
0926 23:17:03.964919 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:03.964922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:03.964967 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:03.965038 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:03.965854 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.966532 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.967636 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.968032 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.968202 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:03.971684 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:03.971689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:03.971692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:03.971717 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:03.974148 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.974759 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.974763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.974765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.974895 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.975233 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.975236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.975238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.975456 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:03.975671 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:03.975833 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 23:17:03.975991 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:17:03.977602 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 22315
I20260926 23:17:03.988404 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:03.978209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.16.62.65:42743)
0926 23:17:03.978373 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:03.978381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:03.978662 (+ 281us) server_negotiation.cc:407] Connection header received
0926 23:17:03.978835 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:03.978839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:03.978884 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:03.978978 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:03.979875 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.980578 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.981709 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:03.982099 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:03.982223 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:03.983981 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:03.983986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:03.983989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:03.984013 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:03.986101 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.986672 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.986679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.986682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.986802 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:03.987144 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:03.987150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:03.987153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:03.987376 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:03.987613 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:03.987906 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 23:17:03.988081 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 23:17:03.989010 22438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:04.071416 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 22315
I20260926 23:17:04.087461 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 22449
Sep 26 23:17:04 dist-test-slave-vcv1 krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:17:04.102094 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:04.088328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.16.62.66:38907)
0926 23:17:04.088921 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:04.088928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:04.088953 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:17:04.089013 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:04.089019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:04.089229 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:04.089356 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:04.090248 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:04.091263 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:04.092546 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:04.093145 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:04.093253 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:04.096839 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:04.096849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:04.096862 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:04.096890 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:04.099620 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:04.100382 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:04.100388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:04.100390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:04.100504 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:04.100900 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:04.100903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:04.100905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:04.101193 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:04.101402 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:04.101645 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 23:17:04.101818 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":301,"threads_started":1}
I20260926 23:17:04.103271 22572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:04.183351 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 22449
I20260926 23:17:04.198645 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 22582
Sep 26 23:17:04 dist-test-slave-vcv1 krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:17:04.214548 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:04.199559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.16.62.67:41113)
0926 23:17:04.200242 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:04.200248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:04.200277 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:17:04.200333 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:04.200340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:04.200544 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:04.200707 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:04.201751 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:04.202742 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:04.203977 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:04.204674 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:04.204836 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:04.208716 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:04.208725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:04.208746 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:04.208779 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:04.211600 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:04.212427 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:04.212435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:04.212438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:04.212580 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:04.212991 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:04.212999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:04.213002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:04.213361 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:04.213708 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:04.213980 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 23:17:04.214240 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":317,"threads_started":1}
I20260926 23:17:04.215849 22705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:04.297750 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 22582
I20260926 23:17:04.313380 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 22222
I20260926 23:17:04.314517 22284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:04.414978 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 22222
2026-09-26T23:17:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3825 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:17:04 dist-test-slave-vcv1 krb5kdc[22737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:17:04 dist-test-slave-vcv1 krb5kdc[22737](info): set up 2 sockets
Sep 26 23:17:04 dist-test-slave-vcv1 krb5kdc[22737](info): commencing operation
krb5kdc: starting...
W20260926 23:17:05.493613 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:17:05 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:17:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:17:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.126.keytab.
Entry for principal oryx/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.126.keytab.
I20260926 23:17:05.689626 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:38973
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.126.keytab
--principal=oryx/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:06.006532 22750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:06.006794 22750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:06.006956 22750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:06.013924 22750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:06.014030 22750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:06.014132 22750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:06.014235 22750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:06.014302 22750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:06.024084 22750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:38973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.126.keytab
--principal=oryx/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:06.025372 22750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:06.026664 22750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:06.037976 22756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:06.038097 22755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:06.038102 22758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:06.039063 22750 server_base.cc:1034] running on GCE node
I20260926 23:17:06.039855 22750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:06.041231 22750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:06.042459 22750 hybrid_clock.cc:648] HybridClock initialized: now 1790464626042414 us; error 37 us; skew 500 ppm
I20260926 23:17:06.042829 22750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:06 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464626, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:06.045790 22750 init.cc:377] Logged in from keytab as oryx/127.16.62.126@KRBTEST.COM (short username oryx)
I20260926 23:17:06.047118 22750 webserver.cc:460] Webserver started at http://127.16.62.126:33485/ using document root <none> and password file <none>
I20260926 23:17:06.047694 22750 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:06.047817 22750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:06.048046 22750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:06.050881 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "144eee699c8e4b32b55bb3f17e2e659d"
format_stamp: "Formatted at 2026-09-26 23:17:06 on dist-test-slave-vcv1"
server_key: "1bd971e7744738743e4f7b2deb1de138"
server_key_iv: "153a90baeb9f18969888928c2ded7859"
server_key_version: "encryptionkey@0"
I20260926 23:17:06.051518 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "144eee699c8e4b32b55bb3f17e2e659d"
format_stamp: "Formatted at 2026-09-26 23:17:06 on dist-test-slave-vcv1"
server_key: "1bd971e7744738743e4f7b2deb1de138"
server_key_iv: "153a90baeb9f18969888928c2ded7859"
server_key_version: "encryptionkey@0"
I20260926 23:17:06.056442 22750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 23:17:06.059449 22765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:06.060755 22750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:06.060941 22750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "144eee699c8e4b32b55bb3f17e2e659d"
format_stamp: "Formatted at 2026-09-26 23:17:06 on dist-test-slave-vcv1"
server_key: "1bd971e7744738743e4f7b2deb1de138"
server_key_iv: "153a90baeb9f18969888928c2ded7859"
server_key_version: "encryptionkey@0"
I20260926 23:17:06.061118 22750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:06.074033 22750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:06.077845 22750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:06.078130 22750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:06.095968 22750 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:38973
I20260926 23:17:06.096050 22816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:38973 every 8 connection(s)
I20260926 23:17:06.097335 22750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:17:06.100940 22817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:06.106822 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 22750
I20260926 23:17:06.107049 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:17:06.107676 16633 external_mini_cluster.cc:1428] Setting key 31f35bcd5e6d125e14655107c137cb12
I20260926 23:17:06.110984 22817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: Bootstrap starting.
I20260926 23:17:06.114466 22817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:06.115485 22817 log.cc:824] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:06.118379 22817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: No bootstrap required, opened a new log
Sep 26 23:17:06 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:06.126884 22817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } }
I20260926 23:17:06.127238 22817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:06.127359 22817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144eee699c8e4b32b55bb3f17e2e659d, State: Initialized, Role: FOLLOWER
I20260926 23:17:06.127884 22817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } }
I20260926 23:17:06.128108 22817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:17:06.128302 22817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:17:06.128482 22817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:06.130267 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:06.109556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.16.62.126:38973)
0926 23:17:06.110286 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:06.110297 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:06.110335 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:17:06.111978 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:06.111999 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:06.112426 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:06.112711 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:06.114494 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:06.115905 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:06.117659 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:06.118264 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:06.118387 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:06.122795 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:06.122802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:06.122816 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:06.122854 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:06.125900 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:06.126548 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:06.126555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:06.126570 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:06.126723 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:06.127080 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:06.127091 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:06.127098 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:06.127764 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:06.128060 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:06.128434 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 23:17:06.128866 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":353,"threads_started":1}
I20260926 23:17:06.131287 22817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } }
I20260926 23:17:06.131774 22817 leader_election.cc:302] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 144eee699c8e4b32b55bb3f17e2e659d; no voters:
I20260926 23:17:06.132416 22817 leader_election.cc:288] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:17:06.132515 22822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:06.133611 22822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 1 LEADER]: Becoming Leader. State: Replica: 144eee699c8e4b32b55bb3f17e2e659d, State: Running, Role: LEADER
I20260926 23:17:06.134141 22822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } }
I20260926 23:17:06.134701 22817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:06.136932 22823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } }
I20260926 23:17:06.137256 22823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:06.138374 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 144eee699c8e4b32b55bb3f17e2e659d. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } }
I20260926 23:17:06.138742 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:06.139905 22828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:17:06.144651 22828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:17:06.153146 22828 catalog_manager.cc:1347] Generated new cluster ID: cc985ffcca974c9cbabddd9fd97737ee
I20260926 23:17:06.153255 22828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:17:06.175315 22828 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:17:06.177136 22828 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:17:06.190204 22828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: Generated new TSK 0
I20260926 23:17:06.191218 22828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab.
Entry for principal oryx/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab.
I20260926 23:17:06.255502 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:38973
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab
--principal=oryx/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:06.574494 22843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:06.574796 22843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:06.574963 22843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:06.582098 22843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:06.582237 22843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:06.582413 22843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:17:06.591677 22843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab
--principal=oryx/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:06.592979 22843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:06.594259 22843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:06.606992 22851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:06.607061 22849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:06.607033 22848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:06.607805 22843 server_base.cc:1034] running on GCE node
I20260926 23:17:06.608407 22843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:06.609252 22843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:06.610443 22843 hybrid_clock.cc:648] HybridClock initialized: now 1790464626610400 us; error 41 us; skew 500 ppm
I20260926 23:17:06.610747 22843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:06 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464626, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:06.613803 22843 init.cc:377] Logged in from keytab as oryx/127.16.62.65@KRBTEST.COM (short username oryx)
I20260926 23:17:06.615170 22843 webserver.cc:460] Webserver started at http://127.16.62.65:43567/ using document root <none> and password file <none>
I20260926 23:17:06.615799 22843 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:06.615895 22843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:06.616127 22843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:06.618992 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "5aba1825270a44f091e6ed417461c3a3"
format_stamp: "Formatted at 2026-09-26 23:17:06 on dist-test-slave-vcv1"
server_key: "07f088eb47d89a79dd2b9d5008d7117c"
server_key_iv: "672f457700d9f7fbf6295e7a61b48ca6"
server_key_version: "encryptionkey@0"
I20260926 23:17:06.619681 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "5aba1825270a44f091e6ed417461c3a3"
format_stamp: "Formatted at 2026-09-26 23:17:06 on dist-test-slave-vcv1"
server_key: "07f088eb47d89a79dd2b9d5008d7117c"
server_key_iv: "672f457700d9f7fbf6295e7a61b48ca6"
server_key_version: "encryptionkey@0"
I20260926 23:17:06.624686 22843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260926 23:17:06.627867 22858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:06.629201 22843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:06.629364 22843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "5aba1825270a44f091e6ed417461c3a3"
format_stamp: "Formatted at 2026-09-26 23:17:06 on dist-test-slave-vcv1"
server_key: "07f088eb47d89a79dd2b9d5008d7117c"
server_key_iv: "672f457700d9f7fbf6295e7a61b48ca6"
server_key_version: "encryptionkey@0"
I20260926 23:17:06.629565 22843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:06.659559 22843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:06.663918 22843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:06.664283 22843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:06.665310 22843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:06.666599 22843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:06.666697 22843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:06.666842 22843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:06.666915 22843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:06.698071 22843 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:35737
I20260926 23:17:06.698153 22970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:35737 every 8 connection(s)
I20260926 23:17:06.699298 22843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:17:06.706425 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 22843
I20260926 23:17:06.706656 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:17:06.707393 16633 external_mini_cluster.cc:1428] Setting key 2ddaa2c16df2b053f701b77a22fd3b56
Sep 26 23:17:06 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464626, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:06.720904 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:06.701721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:54059 (local address 127.16.62.126:38973)
0926 23:17:06.702146 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:06.702152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:06.703155 (+ 1003us) server_negotiation.cc:407] Connection header received
0926 23:17:06.704902 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:06.704906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:06.704969 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:06.705061 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:06.706466 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:06.707587 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:06.709645 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:06.710259 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:06.710412 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:06.714339 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:06.714344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:06.714347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:06.714377 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:06.717763 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:06.718445 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:06.718454 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:06.718460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:06.718608 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:06.719017 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:06.719025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:06.719031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:06.719397 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:06.719543 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:06.720210 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 23:17:06.720448 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":198,"threads_started":1}
I20260926 23:17:06.721406 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:06.702313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:54059)
0926 23:17:06.702992 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:06.703033 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:06.704669 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:06.705270 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:06.705278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:06.705754 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:06.706289 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:06.706304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:06.707731 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:06.707735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:06.709494 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:06.709505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:06.710560 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:06.710565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:06.710793 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:06.711499 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:06.711520 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:06.714146 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:06.717905 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:06.717912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:06.717922 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:06.718314 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:06.718711 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:06.718714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:06.718716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:06.718897 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:06.719507 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:06.719512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:06.719808 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 23:17:06.720007 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 23:17:06.720398 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":292,"threads_started":1}
I20260926 23:17:06.722877 22971 heartbeater.cc:347] Connected to a master server at 127.16.62.126:38973
I20260926 23:17:06.723170 22971 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:06.723872 22971 heartbeater.cc:511] Master 127.16.62.126:38973 requested a full tablet report, sending...
I20260926 23:17:06.725936 22782 ts_manager.cc:194] Registered new tserver with Master: 5aba1825270a44f091e6ed417461c3a3 (127.16.62.65:35737)
I20260926 23:17:06.727311 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:54059
WARNING: no policy specified for oryx/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab.
Entry for principal oryx/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab.
I20260926 23:17:06.753553 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:38973
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab
--principal=oryx/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:07.068574 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:07.068845 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:07.069057 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:07.076496 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:07.076607 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:07.076807 22977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:17:07.086752 22977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab
--principal=oryx/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:07.088111 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:07.089500 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:07.103554 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:07.103977 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:07.103508 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:07.104694 22977 server_base.cc:1034] running on GCE node
I20260926 23:17:07.105365 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:07.106429 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:07.107663 22977 hybrid_clock.cc:648] HybridClock initialized: now 1790464627107593 us; error 56 us; skew 500 ppm
I20260926 23:17:07.107992 22977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:07 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464627, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:07.111756 22977 init.cc:377] Logged in from keytab as oryx/127.16.62.66@KRBTEST.COM (short username oryx)
I20260926 23:17:07.113363 22977 webserver.cc:460] Webserver started at http://127.16.62.66:34413/ using document root <none> and password file <none>
I20260926 23:17:07.114151 22977 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:07.114281 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:07.114570 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:07.117702 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "b0d13abca8344a939dc70777ed62c2f4"
format_stamp: "Formatted at 2026-09-26 23:17:07 on dist-test-slave-vcv1"
server_key: "026e8d6b63b356f261c4f998b82f5826"
server_key_iv: "7905d6c3ed1f4c3692402b0d9f555f37"
server_key_version: "encryptionkey@0"
I20260926 23:17:07.118378 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "b0d13abca8344a939dc70777ed62c2f4"
format_stamp: "Formatted at 2026-09-26 23:17:07 on dist-test-slave-vcv1"
server_key: "026e8d6b63b356f261c4f998b82f5826"
server_key_iv: "7905d6c3ed1f4c3692402b0d9f555f37"
server_key_version: "encryptionkey@0"
I20260926 23:17:07.123327 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:17:07.126557 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:07.127926 22977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:17:07.128099 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "b0d13abca8344a939dc70777ed62c2f4"
format_stamp: "Formatted at 2026-09-26 23:17:07 on dist-test-slave-vcv1"
server_key: "026e8d6b63b356f261c4f998b82f5826"
server_key_iv: "7905d6c3ed1f4c3692402b0d9f555f37"
server_key_version: "encryptionkey@0"
I20260926 23:17:07.128356 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:07.152237 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:07.155989 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:07.156297 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:07.157235 22977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:07.158524 22977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:07.158635 22977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:07.158780 22977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:07.158851 22977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:07.190310 22977 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:42235
I20260926 23:17:07.190341 23104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:42235 every 8 connection(s)
I20260926 23:17:07.191632 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:17:07.201778 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 22977
I20260926 23:17:07.201982 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:17:07.202590 16633 external_mini_cluster.cc:1428] Setting key 2844a74149997cd84beed3b29205720c
Sep 26 23:17:07 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464627, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:07.211476 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:07.194067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:35241 (local address 127.16.62.126:38973)
0926 23:17:07.194233 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:07.194239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:07.195403 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 23:17:07.197238 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:07.197243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:07.197307 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:07.197393 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:07.198700 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:07.199384 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:07.200936 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:07.201358 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:07.201541 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:07.205138 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:07.205146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:07.205154 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:07.205198 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:07.208678 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:07.209328 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:07.209335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:07.209341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:07.209489 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:07.209872 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:07.209878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:07.209883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:07.210204 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:07.210340 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:07.210932 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 23:17:07.211184 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:17:07.212483 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:07.194592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.66:35241)
0926 23:17:07.195234 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:07.195273 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:07.196974 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:07.197580 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:07.197587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:07.198042 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:07.198548 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:07.198563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:07.199506 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:07.199509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:07.200767 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:07.200778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:07.201572 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:07.201579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:07.201846 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:07.202586 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:07.202606 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:07.204937 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:07.208793 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:07.208799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:07.208809 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:07.209203 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:07.209588 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:07.209591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:07.209593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:07.209753 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:07.210311 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:07.210316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:07.210567 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 23:17:07.210766 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 23:17:07.211102 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":248,"threads_started":1}
I20260926 23:17:07.213872 23105 heartbeater.cc:347] Connected to a master server at 127.16.62.126:38973
I20260926 23:17:07.214179 23105 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:07.214818 23105 heartbeater.cc:511] Master 127.16.62.126:38973 requested a full tablet report, sending...
I20260926 23:17:07.216234 22782 ts_manager.cc:194] Registered new tserver with Master: b0d13abca8344a939dc70777ed62c2f4 (127.16.62.66:42235)
I20260926 23:17:07.217473 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:35241
WARNING: no policy specified for oryx/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab.
Entry for principal oryx/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab.
I20260926 23:17:07.247622 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:38973
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab
--principal=oryx/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:07.556056 23110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:07.556378 23110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:07.556475 23110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:07.563362 23110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:07.563457 23110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:07.563575 23110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:17:07.572785 23110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab
--principal=oryx/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:38973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:07.573892 23110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:07.575103 23110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:07.587827 23116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:07.587908 23115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:07.587934 23118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:07.588639 23110 server_base.cc:1034] running on GCE node
I20260926 23:17:07.589167 23110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:07.590041 23110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:07.591240 23110 hybrid_clock.cc:648] HybridClock initialized: now 1790464627591196 us; error 33 us; skew 500 ppm
I20260926 23:17:07.591552 23110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:07 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464627, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:07.594568 23110 init.cc:377] Logged in from keytab as oryx/127.16.62.67@KRBTEST.COM (short username oryx)
I20260926 23:17:07.595924 23110 webserver.cc:460] Webserver started at http://127.16.62.67:44439/ using document root <none> and password file <none>
I20260926 23:17:07.596606 23110 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:07.596733 23110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:07.597048 23110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:07.599781 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "dccd0d1f512249ebaed630f09bbc3c6f"
format_stamp: "Formatted at 2026-09-26 23:17:07 on dist-test-slave-vcv1"
server_key: "1aec6b4d49eacd8feb683dd642f7eee0"
server_key_iv: "6272af526ec3da29f1d884ffe8f4ecf3"
server_key_version: "encryptionkey@0"
I20260926 23:17:07.600533 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "dccd0d1f512249ebaed630f09bbc3c6f"
format_stamp: "Formatted at 2026-09-26 23:17:07 on dist-test-slave-vcv1"
server_key: "1aec6b4d49eacd8feb683dd642f7eee0"
server_key_iv: "6272af526ec3da29f1d884ffe8f4ecf3"
server_key_version: "encryptionkey@0"
I20260926 23:17:07.605305 23110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 23:17:07.608477 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:07.609669 23110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 23:17:07.609826 23110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "dccd0d1f512249ebaed630f09bbc3c6f"
format_stamp: "Formatted at 2026-09-26 23:17:07 on dist-test-slave-vcv1"
server_key: "1aec6b4d49eacd8feb683dd642f7eee0"
server_key_iv: "6272af526ec3da29f1d884ffe8f4ecf3"
server_key_version: "encryptionkey@0"
I20260926 23:17:07.610013 23110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:07.627146 23110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:07.630793 23110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:07.631057 23110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:07.632012 23110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:07.633389 23110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:07.633486 23110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:07.633659 23110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:07.633739 23110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:07.666592 23110 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:41175
I20260926 23:17:07.666633 23237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:41175 every 8 connection(s)
I20260926 23:17:07.667852 23110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:17:07.674530 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 23110
I20260926 23:17:07.674732 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:17:07.675163 16633 external_mini_cluster.cc:1428] Setting key 30c6416763c0e7a5c14217fc68ddc4ca
Sep 26 23:17:07 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464627, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:07.686714 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:07.670209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:54141 (local address 127.16.62.126:38973)
0926 23:17:07.670380 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:07.670384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:07.671534 (+ 1150us) server_negotiation.cc:407] Connection header received
0926 23:17:07.673257 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:07.673262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:07.673355 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:07.673464 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:07.674951 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:07.675795 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:07.677950 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:07.678389 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:07.678484 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:07.681673 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:07.681677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:07.681680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:07.681710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:07.684385 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:07.684951 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:07.684954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:07.684956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:07.685066 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:07.685407 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:07.685410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:07.685412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:07.685659 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:07.685758 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:07.686328 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 23:17:07.686474 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:17:07.687599 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:07.670710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.67:54141)
0926 23:17:07.671366 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:07.671408 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:07.672999 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:07.673695 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:07.673703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:07.674167 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:07.674759 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:07.674802 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:07.675956 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:07.675959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:07.677769 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:07.677783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:07.678571 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:07.678576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:07.678762 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:07.679360 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:07.679383 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:07.681509 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:07.684484 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:07.684489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:07.684500 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:07.684854 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:07.685159 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:07.685162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:07.685164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:07.685319 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:07.685753 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:07.685758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:07.686027 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 23:17:07.686238 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 23:17:07.686568 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":261,"threads_started":1}
I20260926 23:17:07.688979 23238 heartbeater.cc:347] Connected to a master server at 127.16.62.126:38973
I20260926 23:17:07.689282 23238 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:07.690122 23238 heartbeater.cc:511] Master 127.16.62.126:38973 requested a full tablet report, sending...
I20260926 23:17:07.691383 22782 ts_manager.cc:194] Registered new tserver with Master: dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175)
I20260926 23:17:07.692101 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:54141
I20260926 23:17:07.702267 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.62.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.125@KRBTEST.COM" created.
I20260926 23:17:07.730252 22971 heartbeater.cc:503] Master 127.16.62.126:38973 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.125.keytab.
Entry for principal oryx/127.16.62.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.125.keytab.
I20260926 23:17:07.749377 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.125:43067
--webserver_interface=127.16.62.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.125.keytab
--principal=oryx/127.16.62.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:38973,127.16.62.125:43067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:08.066635 23243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:08.066908 23243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:08.067091 23243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:08.074234 23243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:08.074331 23243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:08.074455 23243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:08.074546 23243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:08.074631 23243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:08.084237 23243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:38973,127.16.62.125:43067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.125.keytab
--principal=oryx/127.16.62.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.125:43067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.62.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
W20260926 23:17:08.084584 23243 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.62.126:38973,127.16.62.125:43067'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 23:17:08.085644 23243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:08.086877 23243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:08.098038 23249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:08.098122 23248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:08.098379 23251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:08.099056 23243 server_base.cc:1034] running on GCE node
I20260926 23:17:08.099822 23243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:08.101457 23243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:08.102702 23243 hybrid_clock.cc:648] HybridClock initialized: now 1790464628102635 us; error 45 us; skew 500 ppm
I20260926 23:17:08.103041 23243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:08.106714 23243 init.cc:377] Logged in from keytab as oryx/127.16.62.125@KRBTEST.COM (short username oryx)
I20260926 23:17:08.108525 23243 webserver.cc:460] Webserver started at http://127.16.62.125:37659/ using document root <none> and password file <none>
I20260926 23:17:08.109239 23243 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:08.109360 23243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:08.109746 23243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:08.113967 23243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data/instance:
uuid: "0610f614ebd44277a007bea9a16e8d7b"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "cfdda2ed7288e7d47bbdb041b011c56a"
server_key_iv: "736efe2685291c841d3afa7eb1a3cbde"
server_key_version: "encryptionkey@0"
I20260926 23:17:08.115003 23243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal/instance:
uuid: "0610f614ebd44277a007bea9a16e8d7b"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "cfdda2ed7288e7d47bbdb041b011c56a"
server_key_iv: "736efe2685291c841d3afa7eb1a3cbde"
server_key_version: "encryptionkey@0"
I20260926 23:17:08.119963 23243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 23:17:08.123145 23258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:08.124523 23243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:17:08.124694 23243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal
uuid: "0610f614ebd44277a007bea9a16e8d7b"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "cfdda2ed7288e7d47bbdb041b011c56a"
server_key_iv: "736efe2685291c841d3afa7eb1a3cbde"
server_key_version: "encryptionkey@0"
I20260926 23:17:08.124900 23243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:08.144225 23243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:08.148556 23243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:08.148893 23243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:08.166810 23243 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.125:43067
I20260926 23:17:08.166818 23309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.125:43067 every 8 connection(s)
I20260926 23:17:08.168061 23243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
I20260926 23:17:08.171849 23310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:08.172399 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 23243
I20260926 23:17:08.181347 23310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.125@KRBTEST.COM
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.125@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:08.196334 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.174595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.16.62.125:43067)
0926 23:17:08.175376 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.175388 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.175426 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:17:08.176881 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.176900 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.177282 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.177587 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.179298 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.180647 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.182388 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.183045 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.183161 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.187802 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.187808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.187820 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.187852 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.191666 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.192481 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.192490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.192499 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.192677 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.193140 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.193146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.193152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.193773 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.194057 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.194506 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.195139 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"spinlock_wait_cycles":14592,"thread_start_us":390,"threads_started":1}
I20260926 23:17:08.199389 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.184339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.16.62.126:38973)
0926 23:17:08.184545 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.184552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.185239 (+ 687us) server_negotiation.cc:407] Connection header received
0926 23:17:08.185396 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.185402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.185495 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.185608 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.186660 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.187402 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.189061 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.189766 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.189889 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.193849 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.193856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.193861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.193903 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.196886 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.197584 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.197589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.197592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.197748 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.198119 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.198124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.198127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.198431 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.198583 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.199033 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.199184 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:17:08.199496 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.184339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:41888)
0926 23:17:08.185077 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.185131 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.185273 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.185663 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.185667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.186194 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.186511 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.186522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.187536 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.187540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.188920 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.188930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.190164 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.190170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.190364 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.191017 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.191045 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.193675 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.197028 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.197040 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.197046 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.197457 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.197859 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.197864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.197868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.197993 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.198555 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.198562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.198734 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.198949 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.199198 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":383,"threads_started":1}
I20260926 23:17:08.201197 23310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.125@KRBTEST.COM for oryx/127.16.62.125@KRBTEST.COM
I20260926 23:17:08.212359 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.201838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.0.0.1:54942)
0926 23:17:08.202028 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.202046 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.202135 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.202396 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.202400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.202733 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.202843 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.202851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.203766 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.203772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.204665 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.204675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.205319 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.205324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.205513 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.205969 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.205984 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.207759 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.210460 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.210463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.210465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.210837 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.211126 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.211129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.211140 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.211252 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.211666 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.211669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.211778 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.211878 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.212057 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1}
I20260926 23:17:08.212514 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.201966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.16.62.125:43067)
0926 23:17:08.202105 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.202110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.202126 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:17:08.202225 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.202230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.202286 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.202383 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.202966 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.203630 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.204795 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.205183 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.205312 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.207897 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.207902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.207905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.207930 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.210336 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.210928 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.210931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.210933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.211040 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.211336 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.211339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.211341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.211575 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.211672 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.211951 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.212225 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":46}
I20260926 23:17:08.216939 23310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: Bootstrap starting.
I20260926 23:17:08.219811 23310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:08.219856 23105 heartbeater.cc:503] Master 127.16.62.126:38973 was elected leader, sending a full tablet report...
I20260926 23:17:08.221074 23310 log.cc:824] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:08.223588 23310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: No bootstrap required, opened a new log
I20260926 23:17:08.230474 23310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } }
I20260926 23:17:08.230783 23310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:08.230875 23310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0610f614ebd44277a007bea9a16e8d7b, State: Initialized, Role: FOLLOWER
I20260926 23:17:08.231489 23310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } }
I20260926 23:17:08.232525 23316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } } }
I20260926 23:17:08.232893 23316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:17:08.233491 23310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:17:08.241988 23327 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:17:08.242113 23327 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:17:08.244349 23243 master_runner.cc:428] Detected that this master 0610f614ebd44277a007bea9a16e8d7b is joining an existing cluster
I20260926 23:17:08.244423 23243 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.62.125:43067
I20260926 23:17:08.264019 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.247923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.16.62.126:38973)
0926 23:17:08.248097 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.248105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.248984 (+ 879us) server_negotiation.cc:407] Connection header received
0926 23:17:08.249229 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.249234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.249334 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.249425 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.250760 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.251647 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.253621 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.254261 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.254360 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.256895 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.256903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.256909 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.256948 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.260566 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.261443 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.261452 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.261458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.261611 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.262062 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.262070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.262075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.262473 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.262609 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.263310 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.263563 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 23:17:08.265712 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.248060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:41904)
0926 23:17:08.248876 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.248898 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.249096 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.249437 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.249441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.250165 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.250607 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.250621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.251779 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.251786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.253454 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.253475 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.254639 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.254645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.254831 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.255539 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.255562 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.256688 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.260718 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.260723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.260732 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.261298 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.261743 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.261750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.261756 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.261937 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.262690 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.262698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.262938 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.264616 (+ 1678us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.265023 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":32,"thread_start_us":443,"threads_started":1}
I20260926 23:17:08.267297 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.248300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.0.0.1:54948)
0926 23:17:08.249235 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.249253 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.249374 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.249834 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.249838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.250530 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.251017 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.251027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.252537 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.252545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.254298 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.254316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.258478 (+ 4162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.258485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.258735 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.259294 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.259317 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.260311 (+ 994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.264467 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.264472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.264476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.265039 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.265429 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.265433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.265437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.265585 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.266272 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.266277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.266415 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.266547 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.266822 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":190,"thread_start_us":546,"threads_started":1}
I20260926 23:17:08.268262 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.247953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.16.62.125:43067)
0926 23:17:08.248106 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.248113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.249338 (+ 1225us) server_negotiation.cc:407] Connection header received
0926 23:17:08.249498 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.249502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.249580 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.249643 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.251164 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.252317 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.254439 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.255259 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.255363 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.260489 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.260498 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.260505 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.260543 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.264327 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.265159 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.265164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.265168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.265309 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.265703 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.265709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.265716 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.266061 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.266186 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.267702 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.267905 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for oryx/127.16.62.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.124@KRBTEST.COM" created.
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464626, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.126@KRBTEST.COM for oryx/127.16.62.125@KRBTEST.COM
I20260926 23:17:08.290946 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.272912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.0.0.1:54954)
0926 23:17:08.273664 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.273712 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.273920 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.274422 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.274429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.275255 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.275516 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.275529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.276607 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.276613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.278381 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.278396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.279678 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.279684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.279888 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.280651 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.280674 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.283538 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.287793 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.287807 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.287815 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.288415 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.288839 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.288845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.288850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.289020 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.289733 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.289742 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.289936 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.290156 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.290507 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":393,"threads_started":1}
I20260926 23:17:08.290995 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.273022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.16.62.125:43067)
0926 23:17:08.273183 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.273189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.273791 (+ 602us) server_negotiation.cc:407] Connection header received
0926 23:17:08.274143 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.274149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.274261 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.274363 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.275724 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.276483 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.278538 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.279318 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.279411 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.283740 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.283750 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.283757 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.283798 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.287643 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.288534 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.288542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.288548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.288712 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.289141 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.289149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.289155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.289586 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.289735 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.290279 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.290526 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:17:08.293720 22782 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.62.125:43067: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b"
member_type: NON_VOTER
last_known_addr {
host: "127.16.62.125"
port: 43067
}
attrs {
promote: true
}
}
dest_uuid: "144eee699c8e4b32b55bb3f17e2e659d"
cas_config_opid_index: -1
I20260926 23:17:08.295318 22782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } }
I20260926 23:17:08.298808 23340 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0610f614ebd44277a007bea9a16e8d7b (127.16.62.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } } }
I20260926 23:17:08.299880 23285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:08.302965 23340 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.62.125:43067
I20260926 23:17:08.303666 23243 master.cc:537] Master@127.16.62.125:43067 shutting down...
I20260926 23:17:08.303628 23339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } } }
I20260926 23:17:08.304790 23339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:08.305423 23285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Refusing update from remote peer 144eee699c8e4b32b55bb3f17e2e659d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 23:17:08.306516 23340 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.124.keytab.
Entry for principal oryx/127.16.62.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.124.keytab.
W20260926 23:17:08.309803 22769 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 23:17:08.309809 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.124:46245
--webserver_interface=127.16.62.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.124.keytab
--principal=oryx/127.16.62.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:38973,127.16.62.125:43067,127.16.62.124:46245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:08.316792 22769 consensus_peers.cc:487] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d -> Peer 0610f614ebd44277a007bea9a16e8d7b (127.16.62.125:43067): Couldn't send request to peer 0610f614ebd44277a007bea9a16e8d7b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 23:17:08.322804 23243 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:08.323194 23243 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:08.323386 23243 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: stopping tablet replica
I20260926 23:17:08.328943 23243 master.cc:552] Master@127.16.62.125:43067 shutdown complete.
I20260926 23:17:08.329131 23243 master_runner.cc:305] Clearing existing system tablet
I20260926 23:17:08.329551 23243 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:08.329656 23243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:08.337837 23243 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.009s
I20260926 23:17:08.339365 23345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:08.339665 23243 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:08.339937 23243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 23:17:08.340020 23243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal
uuid: "0610f614ebd44277a007bea9a16e8d7b"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "cfdda2ed7288e7d47bbdb041b011c56a"
server_key_iv: "736efe2685291c841d3afa7eb1a3cbde"
server_key_version: "encryptionkey@0"
I20260926 23:17:08.343915 23243 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:08.348515 23243 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 23:17:08.348688 23243 log.cc:1196] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 23:17:08.349043 23243 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: Deleting consensus metadata
I20260926 23:17:08.349453 23243 master_runner.cc:315] Copying system tablet from 127.16.62.126:38973
I20260926 23:17:08.352345 23243 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.62.126:38973
I20260926 23:17:08.364612 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.353214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.16.62.126:38973)
0926 23:17:08.353438 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.353448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.353904 (+ 456us) server_negotiation.cc:407] Connection header received
0926 23:17:08.354106 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.354112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.354200 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.354326 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.355220 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.356004 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.357278 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.357689 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.357822 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.359497 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.359510 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.359514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.359545 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.362093 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.362680 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.362686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.362690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.362807 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.363172 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.363177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.363180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.363430 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.363584 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.363966 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.364213 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:17:08.364688 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.353150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:41912)
0926 23:17:08.353795 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.353810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.353973 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.354491 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.354494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.354866 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.355070 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.355079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.356173 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.356179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.357119 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.357131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.357838 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.357844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.358011 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.358666 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.358683 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.359306 (+ 623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.362224 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.362229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.362233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.362564 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.362916 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.362921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.362925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.363060 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.363563 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.363568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.363726 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.363865 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.364178 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":425,"threads_started":1}
I20260926 23:17:08.365829 22802 tablet_copy_service.cc:140] P 144eee699c8e4b32b55bb3f17e2e659d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0610f614ebd44277a007bea9a16e8d7b ({username='oryx', principal='oryx/127.16.62.125@KRBTEST.COM'} at 127.0.0.1:41912)
I20260926 23:17:08.366231 22802 tablet_copy_service.cc:161] P 144eee699c8e4b32b55bb3f17e2e659d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0610f614ebd44277a007bea9a16e8d7b at {username='oryx', principal='oryx/127.16.62.125@KRBTEST.COM'} at 127.0.0.1:41912: session id = 0610f614ebd44277a007bea9a16e8d7b-00000000000000000000000000000000
I20260926 23:17:08.369446 22802 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 23:17:08.371381 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 23:17:08.375938 23243 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: tablet copy: Starting download of 0 data blocks...
I20260926 23:17:08.376287 23243 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: tablet copy: Starting download of 1 WAL segments...
I20260926 23:17:08.378935 23243 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 23:17:08.381971 22802 tablet_copy_service.cc:342] P 144eee699c8e4b32b55bb3f17e2e659d: Request end of tablet copy session 0610f614ebd44277a007bea9a16e8d7b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.62.125@KRBTEST.COM'} at 127.0.0.1:41912
I20260926 23:17:08.382148 22802 tablet_copy_service.cc:433] P 144eee699c8e4b32b55bb3f17e2e659d: ending tablet copy session 0610f614ebd44277a007bea9a16e8d7b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0610f614ebd44277a007bea9a16e8d7b
I20260926 23:17:08.392356 23243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:08.395002 23352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:08.397456 23353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:08.405311 23243 server_base.cc:1034] running on GCE node
W20260926 23:17:08.405699 23355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:08.406070 23243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:08.406664 23243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:08.407910 23243 hybrid_clock.cc:648] HybridClock initialized: now 1790464628407855 us; error 32 us; skew 500 ppm
I20260926 23:17:08.408123 23243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:08.410024 23243 init.cc:377] Logged in from keytab as oryx/127.16.62.125@KRBTEST.COM (short username oryx)
I20260926 23:17:08.411180 23243 webserver.cc:460] Webserver started at http://127.16.62.125:39283/ using document root <none> and password file <none>
I20260926 23:17:08.411496 23243 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:08.411589 23243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:08.414271 23243 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 23:17:08.415885 23362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:08.416308 23243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 23:17:08.416450 23243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal
uuid: "0610f614ebd44277a007bea9a16e8d7b"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "cfdda2ed7288e7d47bbdb041b011c56a"
server_key_iv: "736efe2685291c841d3afa7eb1a3cbde"
server_key_version: "encryptionkey@0"
I20260926 23:17:08.416617 23243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:08.427677 23243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:08.431319 23243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:08.462495 23243 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.125:43067
I20260926 23:17:08.462522 23413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.125:43067 every 8 connection(s)
I20260926 23:17:08.463400 23243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
I20260926 23:17:08.470252 23414 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:17:08.472972 23414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: Bootstrap starting.
I20260926 23:17:08.486143 23414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:17:08.486608 23414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: Bootstrap complete.
I20260926 23:17:08.487485 23414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } }
I20260926 23:17:08.487739 23414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0610f614ebd44277a007bea9a16e8d7b, State: Initialized, Role: LEARNER
I20260926 23:17:08.487993 23414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } }
I20260926 23:17:08.488880 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } } }
I20260926 23:17:08.489241 23414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:08.489270 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: This master's current role is: LEARNER
I20260926 23:17:08.499017 23428 catalog_manager.cc:1259] Loaded cluster ID: cc985ffcca974c9cbabddd9fd97737ee
I20260926 23:17:08.499137 23428 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: loading cluster ID for follower catalog manager: success
I20260926 23:17:08.501662 23428 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: acquiring CA information for follower catalog manager: success
I20260926 23:17:08.503576 23428 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:17:08.694393 23238 heartbeater.cc:503] Master 127.16.62.126:38973 was elected leader, sending a full tablet report...
W20260926 23:17:08.725706 23343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:08.725972 23343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:08.726122 23343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:08.733731 23343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:08.733856 23343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:08.733989 23343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:08.734099 23343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:08.734172 23343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:08.744704 23343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:38973,127.16.62.125:43067,127.16.62.124:46245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.124.keytab
--principal=oryx/127.16.62.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.124:46245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.62.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:08.746104 23343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:08.747473 23343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:08.759375 23433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:08.759441 23436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:08.759852 23434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:08.760280 23343 server_base.cc:1034] running on GCE node
I20260926 23:17:08.760939 23343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:08.762190 23343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:08.763417 23343 hybrid_clock.cc:648] HybridClock initialized: now 1790464628763375 us; error 33 us; skew 500 ppm
I20260926 23:17:08.763757 23343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:08.767040 23343 init.cc:377] Logged in from keytab as oryx/127.16.62.124@KRBTEST.COM (short username oryx)
I20260926 23:17:08.768486 23343 webserver.cc:460] Webserver started at http://127.16.62.124:35777/ using document root <none> and password file <none>
I20260926 23:17:08.769160 23343 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:08.769265 23343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:08.769501 23343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:08.772305 23343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data/instance:
uuid: "93301179318146f7a4aec1331bd3df3f"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "ca3153c7478db32412376e06403a5744"
server_key_iv: "acfdb9af312791943617fd3ffc0a32f0"
server_key_version: "encryptionkey@0"
I20260926 23:17:08.773334 23343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal/instance:
uuid: "93301179318146f7a4aec1331bd3df3f"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "ca3153c7478db32412376e06403a5744"
server_key_iv: "acfdb9af312791943617fd3ffc0a32f0"
server_key_version: "encryptionkey@0"
I20260926 23:17:08.778944 23343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:17:08.782025 23443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:08.783301 23343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:17:08.783469 23343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal
uuid: "93301179318146f7a4aec1331bd3df3f"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "ca3153c7478db32412376e06403a5744"
server_key_iv: "acfdb9af312791943617fd3ffc0a32f0"
server_key_version: "encryptionkey@0"
I20260926 23:17:08.783648 23343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:08.806658 23343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:08.810349 23343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:08.810559 23343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:08.828507 23343 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.124:46245
I20260926 23:17:08.828644 23494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.124:46245 every 8 connection(s)
I20260926 23:17:08.829811 23343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
I20260926 23:17:08.833529 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:08.839051 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 23343
I20260926 23:17:08.840834 23495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.124@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.124@KRBTEST.COM
I20260926 23:17:08.864157 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.843595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.16.62.126:38973)
0926 23:17:08.843751 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.843755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.844445 (+ 690us) server_negotiation.cc:407] Connection header received
0926 23:17:08.844834 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.844840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.844920 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.845023 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.846138 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.846918 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.848512 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.848881 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.849006 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.857847 (+ 8841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.857852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.857855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.857886 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.861142 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.861915 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.861923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.861926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.862075 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.862521 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.862525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.862527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.862799 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.862954 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.863604 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.863820 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:17:08.865339 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.843603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:41914)
0926 23:17:08.844310 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.844352 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.844609 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.845081 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.845087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.845657 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.845984 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.845998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.847071 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.847076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.848369 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.848380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.849092 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.849098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.849331 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.855380 (+ 6049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.855401 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.857644 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.861268 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.861276 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.861288 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.861773 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.862198 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.862209 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.862212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.862373 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.862931 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.862938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.863241 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.863494 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.863961 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":71,"thread_start_us":326,"threads_started":1}
I20260926 23:17:08.865376 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.841316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.16.62.124:46245)
0926 23:17:08.842103 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.842114 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.842164 (+ 50us) server_negotiation.cc:407] Connection header received
0926 23:17:08.844293 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.844315 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.844652 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.844876 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.846229 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.847375 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.848625 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.849118 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.849331 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.858956 (+ 9625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.858966 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.858981 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.859017 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.862225 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.862851 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.862859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.862863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.862994 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.863364 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.863370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.863373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.863669 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.863969 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.864321 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.864543 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":574,"thread_start_us":345,"threads_started":1}
I20260926 23:17:08.866779 23495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.124@KRBTEST.COM for oryx/127.16.62.125@KRBTEST.COM
I20260926 23:17:08.878768 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.867550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.0.0.1:54964)
0926 23:17:08.867744 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.867766 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.867883 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.868433 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.868436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.868710 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.868835 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.868842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.870053 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.870056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.870952 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.870962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.871579 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.871582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.871721 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.872300 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.872315 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.874170 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.876819 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.876823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.876826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.877181 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.877498 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.877501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.877503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.877615 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.878127 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.878130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.878254 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.878351 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.878558 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 23:17:08.878793 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.867568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.16.62.125:43067)
0926 23:17:08.868119 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.868125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.868184 (+ 59us) server_negotiation.cc:407] Connection header received
0926 23:17:08.868244 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.868250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.868317 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.868399 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.868984 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.869936 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.871066 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.871459 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.871547 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.874390 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.874413 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.874420 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.874456 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.876708 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.877290 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.877294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.877296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.877410 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.877728 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.877732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.877735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.877999 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.878156 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.878426 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.878583 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":272,"threads_started":1}
I20260926 23:17:08.879818 23495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464628, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.124@KRBTEST.COM for oryx/127.16.62.124@KRBTEST.COM
I20260926 23:17:08.891059 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.880396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.0.0.1:44396)
0926 23:17:08.880539 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.880561 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.880649 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.880933 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.880936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.881219 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.881344 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.881350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.882158 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.882161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.882947 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.882956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.883579 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.883582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.883728 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.884295 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.884310 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.886036 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.888730 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.888734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.888736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.889045 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.889394 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.889398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.889430 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.889609 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.890082 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.890085 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.890188 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.890330 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.890589 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 23:17:08.891068 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.880579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.16.62.124:46245)
0926 23:17:08.880723 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.880727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.880740 (+ 13us) server_negotiation.cc:407] Connection header received
0926 23:17:08.880774 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.880777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.880821 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.880905 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.881448 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.882065 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.883069 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.883463 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.883556 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.886200 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.886204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.886207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.886231 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.888612 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.889147 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.889150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.889152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.889283 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.889721 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.889725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.889727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.889960 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.890111 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.890418 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.890706 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":58}
I20260926 23:17:08.897157 23495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Bootstrap starting.
I20260926 23:17:08.898686 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.892640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.16.62.125:43067)
0926 23:17:08.892855 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.892860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.893158 (+ 298us) server_negotiation.cc:407] Connection header received
0926 23:17:08.893331 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.893334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.893419 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.893506 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:08.893993 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.894728 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.897009 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.897867 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.897976 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.898134 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.898429 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.898523 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:17:08.898881 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.892391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.0.0.1:54980)
0926 23:17:08.893056 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.893077 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.893215 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.893576 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.893581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.893610 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:08.893850 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.893860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.894869 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.894875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.896845 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.896859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.898053 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.898072 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.898243 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.898294 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.898391 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.898519 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":388,"threads_started":1}
I20260926 23:17:08.899880 23389 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 23:17:08.900458 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 144eee699c8e4b32b55bb3f17e2e659d. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } } }
I20260926 23:17:08.900712 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: This master's current role is: LEARNER
I20260926 23:17:08.900938 23495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:08.901971 23495 log.cc:824] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:08.904485 23495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: No bootstrap required, opened a new log
I20260926 23:17:08.911407 23495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } }
I20260926 23:17:08.911715 23495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:08.911847 23495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93301179318146f7a4aec1331bd3df3f, State: Initialized, Role: FOLLOWER
I20260926 23:17:08.912443 23495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } }
I20260926 23:17:08.913300 23504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } } }
I20260926 23:17:08.913565 23504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:17:08.914281 23495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:08.920089 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 22843
W20260926 23:17:08.926677 23515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:17:08.926906 23515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:17:08.927657 23343 master_runner.cc:428] Detected that this master 93301179318146f7a4aec1331bd3df3f is joining an existing cluster
I20260926 23:17:08.927765 23343 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.62.124:46245
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.65@KRBTEST.COM
I20260926 23:17:08.942974 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.921375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.16.62.65:35737)
0926 23:17:08.921988 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.921996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.922024 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:17:08.922090 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.922098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.922354 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.922527 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.924003 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.925595 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.927394 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.927909 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.928042 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.933120 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.933134 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.933148 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.933189 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.937942 (+ 4753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.939037 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.939049 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.939055 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.939231 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.939810 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.939817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.939823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.940365 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.940779 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.942016 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.942345 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":672,"threads_started":1}
I20260926 23:17:08.944970 22966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:08.958257 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.931973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.16.62.126:38973)
0926 23:17:08.932212 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.932218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.940812 (+ 8594us) server_negotiation.cc:407] Connection header received
0926 23:17:08.941108 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.941117 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.941203 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.941288 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.943352 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.944466 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.946604 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.947272 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.947391 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.949919 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.949926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.949930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.949970 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.953439 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.954200 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.954205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.954208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.954363 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.956043 (+ 1680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.956051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.956057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.956461 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.956610 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.957667 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.957822 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:17:08.959360 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.932010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:41924)
0926 23:17:08.940679 (+ 8669us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.940701 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.940945 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.941544 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.941550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.942204 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.943220 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.943234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.944661 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.944667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.946431 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.946446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.947623 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.947629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.947809 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.948529 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.948552 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.949721 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.953596 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.953601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.953605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.954007 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.954491 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.954496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.954499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.955919 (+ 1420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.956665 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.956672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.956915 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.957089 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.957446 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8502,"thread_start_us":8338,"threads_started":1}
I20260926 23:17:08.976168 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.933615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.16.62.125:43067)
0926 23:17:08.933781 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.933787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.941324 (+ 7537us) server_negotiation.cc:407] Connection header received
0926 23:17:08.941669 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.941675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.941754 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.941849 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.952918 (+ 11069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.954013 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.963422 (+ 9409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.964215 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.964325 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.968316 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.968324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.968329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.968372 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.972049 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.972948 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.972954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.972959 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.973122 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.973641 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.973647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.973651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.974030 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.974174 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.975011 (+ 837us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.975443 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:17:08.978099 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.933839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.0.0.1:54990)
0926 23:17:08.941080 (+ 7241us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.941096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.941528 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.942204 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.942209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.952352 (+ 10143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.952722 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.952734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.954192 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.954196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.960354 (+ 6158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.960365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.964612 (+ 4247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.964618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.964782 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.965833 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.965855 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.967514 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.972263 (+ 4749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.972267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.972271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.972801 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.973293 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.973297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.973300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.973501 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.974267 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.974271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.974593 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.976576 (+ 1983us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.977101 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7109,"thread_start_us":6943,"threads_started":1}
Sep 26 23:17:08 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464626, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.126@KRBTEST.COM for oryx/127.16.62.124@KRBTEST.COM
I20260926 23:17:08.985925 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.941248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.16.62.124:46245)
0926 23:17:08.941456 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.941463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.956551 (+ 15088us) server_negotiation.cc:407] Connection header received
0926 23:17:08.956766 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.956770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.956842 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.956907 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.960503 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.961561 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.968101 (+ 6540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.968863 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.969237 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.976076 (+ 6839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.976082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.976086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.976157 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.982196 (+ 6039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.983102 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.983107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.983111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.983321 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.983739 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.983744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.983747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.984308 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.984492 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.985112 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.985390 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":103}
I20260926 23:17:08.987258 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.941254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.0.0.1:44406)
0926 23:17:08.956467 (+ 15213us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.956490 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.956628 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.957063 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.957068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.958731 (+ 1663us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.960008 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.960022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.961786 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.961790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.967954 (+ 6164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.967967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.969516 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.969523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.969693 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.971895 (+ 2202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.971916 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.973048 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.982348 (+ 9300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.982353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.982357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.982991 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.983443 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.983447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.983451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.983635 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.984556 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.984560 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.984779 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.986076 (+ 1297us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.986428 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15063,"thread_start_us":14935,"threads_started":1}
I20260926 23:17:08.997421 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.965948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.0.0.1:44416)
0926 23:17:08.966119 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:08.966138 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:08.966239 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:08.967264 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:08.967269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:08.967698 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:08.967868 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.967879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.973214 (+ 5335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.973221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.974689 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:08.974705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.978257 (+ 3552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.978263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:08.978553 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.981195 (+ 2642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:08.981231 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:08.985426 (+ 4195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:08.994074 (+ 8648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.994080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.994085 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.994785 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.995388 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:08.995393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:08.995397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:08.995635 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:08.996487 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:08.996493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:08.996683 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 23:17:08.996825 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 23:17:08.997091 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 23:17:08.997934 23526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:08.966241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.16.62.124:46245)
0926 23:17:08.966854 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:08.966860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:08.966882 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:17:08.966963 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:08.966969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:08.967064 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:08.968008 (+ 944us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:08.970284 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.971506 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.974896 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:08.975869 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:08.985591 (+ 9722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:08.986392 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:08.986403 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:08.986419 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:08.986454 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:08.993902 (+ 7448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.994994 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.995000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.995003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.995254 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:08.995776 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:08.995781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:08.995785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:08.996346 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:08.996508 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:08.997004 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 23:17:08.997216 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1915,"server-negotiator.queue_time_us":488,"thread_start_us":328,"threads_started":1}
I20260926 23:17:08.999392 22782 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.62.124:46245: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "93301179318146f7a4aec1331bd3df3f"
member_type: NON_VOTER
last_known_addr {
host: "127.16.62.124"
port: 46245
}
attrs {
promote: true
}
}
dest_uuid: "144eee699c8e4b32b55bb3f17e2e659d"
cas_config_opid_index: 5
I20260926 23:17:09.000841 22782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } }
I20260926 23:17:09.003177 23501 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 93301179318146f7a4aec1331bd3df3f (127.16.62.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.009927 23470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:09.016948 23470 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Refusing update from remote peer 144eee699c8e4b32b55bb3f17e2e659d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 23:17:09.018379 23501 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.62.124:46245
I20260926 23:17:09.019016 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.019757 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:09.020587 23528 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 23:17:09.022531 23343 master.cc:537] Master@127.16.62.124:46245 shutting down...
I20260926 23:17:09.024565 23389 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 LEARNER]: Refusing update from remote peer 144eee699c8e4b32b55bb3f17e2e659d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 23:17:09.025561 23501 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 23:17:09.029024 23389 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 93301179318146f7a4aec1331bd3df3f (127.16.62.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.035120 23343 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:09.035089 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: NON_VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: true } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.035424 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: This master's current role is: LEARNER
I20260926 23:17:09.035722 23343 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:09.035987 23343 tablet_replica.cc:330] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: stopping tablet replica
I20260926 23:17:09.037022 23501 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: attempting to promote NON_VOTER 0610f614ebd44277a007bea9a16e8d7b to VOTER
I20260926 23:17:09.040355 23501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } }
I20260926 23:17:09.049222 23389 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 LEARNER]: Refusing update from remote peer 144eee699c8e4b32b55bb3f17e2e659d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 23:17:09.050375 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.049955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.0.0.1:44432)
0926 23:17:09.050091 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:09.050196 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:46245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 23:17:09.051002 23528 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 23:17:09.051338 23343 master.cc:552] Master@127.16.62.124:46245 shutdown complete.
I20260926 23:17:09.051460 23343 master_runner.cc:305] Clearing existing system tablet
I20260926 23:17:09.051616 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.051276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.0.0.1:44442)
0926 23:17:09.051412 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:09.051502 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:46245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 23:17:09.052124 22769 consensus_peers.cc:487] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d -> Peer 93301179318146f7a4aec1331bd3df3f (127.16.62.124:46245): Couldn't send request to peer 93301179318146f7a4aec1331bd3df3f. Status: Network error: Client connection negotiation failed: client connection to 127.16.62.124:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 23:17:09.052003 23343 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:09.052793 23343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:09.053915 23527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0610f614ebd44277a007bea9a16e8d7b (127.16.62.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.058758 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.059039 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:09.059829 23528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.060060 23528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:09.060883 23389 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0610f614ebd44277a007bea9a16e8d7b (127.16.62.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.066792 23343 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.008s sys 0.004s
I20260926 23:17:09.067245 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.067622 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:17:09.069036 23534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:09.069316 23343 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:09.069561 23343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 23:17:09.069689 23343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal
uuid: "93301179318146f7a4aec1331bd3df3f"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "ca3153c7478db32412376e06403a5744"
server_key_iv: "acfdb9af312791943617fd3ffc0a32f0"
server_key_version: "encryptionkey@0"
I20260926 23:17:09.074364 23343 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:09.081039 23343 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 23:17:09.081174 23343 log.cc:1196] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 23:17:09.081578 23343 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Deleting consensus metadata
I20260926 23:17:09.081812 23343 master_runner.cc:315] Copying system tablet from 127.16.62.126:38973
I20260926 23:17:09.085588 23343 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: tablet copy: Beginning tablet copy session from remote peer at address 127.16.62.126:38973
I20260926 23:17:09.104622 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.086323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:41926)
0926 23:17:09.088550 (+ 2227us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:09.088574 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:09.088722 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:09.089279 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:09.089283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:09.089798 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:09.090056 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.090067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.091551 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.091557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.093137 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.093150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.094193 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.094198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.094384 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.095102 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:09.095123 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:09.096197 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:09.100940 (+ 4743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:09.100948 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:09.100955 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:09.101468 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:09.102029 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:09.102036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:09.102042 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:09.102235 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:09.103125 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:09.103133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:09.103356 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 23:17:09.103526 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 23:17:09.103965 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"thread_start_us":1956,"threads_started":1}
I20260926 23:17:09.106148 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.086359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.16.62.126:38973)
0926 23:17:09.086571 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:09.086577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:09.088693 (+ 2116us) server_negotiation.cc:407] Connection header received
0926 23:17:09.088909 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:09.088915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:09.089010 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:09.089082 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:09.090243 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.091408 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.093280 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.094019 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.094140 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.096427 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:09.096434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:09.096439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:09.096475 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:09.100702 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.101639 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.101648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.101654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.101865 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.102371 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.102378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.102382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.102853 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:09.103029 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:09.105226 (+ 2197us) server_negotiation.cc:299] Negotiation successful
0926 23:17:09.105553 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 23:17:09.107204 22802 tablet_copy_service.cc:140] P 144eee699c8e4b32b55bb3f17e2e659d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 93301179318146f7a4aec1331bd3df3f ({username='oryx', principal='oryx/127.16.62.124@KRBTEST.COM'} at 127.0.0.1:41926)
I20260926 23:17:09.107359 22802 tablet_copy_service.cc:161] P 144eee699c8e4b32b55bb3f17e2e659d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 93301179318146f7a4aec1331bd3df3f at {username='oryx', principal='oryx/127.16.62.124@KRBTEST.COM'} at 127.0.0.1:41926: session id = 93301179318146f7a4aec1331bd3df3f-00000000000000000000000000000000
I20260926 23:17:09.108431 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 22843
I20260926 23:17:09.111481 22802 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 23:17:09.113389 23343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 23:17:09.121616 23343 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: tablet copy: Starting download of 0 data blocks...
I20260926 23:17:09.121925 23343 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: tablet copy: Starting download of 1 WAL segments...
I20260926 23:17:09.124399 23343 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 23:17:09.128710 22802 tablet_copy_service.cc:342] P 144eee699c8e4b32b55bb3f17e2e659d: Request end of tablet copy session 93301179318146f7a4aec1331bd3df3f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.62.124@KRBTEST.COM'} at 127.0.0.1:41926
I20260926 23:17:09.128855 22802 tablet_copy_service.cc:433] P 144eee699c8e4b32b55bb3f17e2e659d: ending tablet copy session 93301179318146f7a4aec1331bd3df3f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 93301179318146f7a4aec1331bd3df3f
I20260926 23:17:09.131209 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:35737
--local_ip_for_outbound_sockets=127.16.62.65
--tserver_master_addrs=127.16.62.126:38973
--webserver_port=43567
--webserver_interface=127.16.62.65
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab
--principal=oryx/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.62.126:38973,127.16.62.125:43067
--tserver_master_addrs=127.16.62.126:38973,127.16.62.125:43067,127.16.62.124:46245 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:17:09.141533 23343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:09.144120 23542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:09.148277 23543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:09.155064 23545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:09.155953 23343 server_base.cc:1034] running on GCE node
I20260926 23:17:09.156316 23343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:09.157048 23343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:09.158300 23343 hybrid_clock.cc:648] HybridClock initialized: now 1790464629158256 us; error 36 us; skew 500 ppm
I20260926 23:17:09.158540 23343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:09 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:09.160657 23343 init.cc:377] Logged in from keytab as oryx/127.16.62.124@KRBTEST.COM (short username oryx)
I20260926 23:17:09.165127 23343 webserver.cc:460] Webserver started at http://127.16.62.124:46143/ using document root <none> and password file <none>
I20260926 23:17:09.165439 23343 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:09.165544 23343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:09.168587 23343 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 23:17:09.170428 23552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:09.170838 23343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 23:17:09.170987 23343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal
uuid: "93301179318146f7a4aec1331bd3df3f"
format_stamp: "Formatted at 2026-09-26 23:17:08 on dist-test-slave-vcv1"
server_key: "ca3153c7478db32412376e06403a5744"
server_key_iv: "acfdb9af312791943617fd3ffc0a32f0"
server_key_version: "encryptionkey@0"
I20260926 23:17:09.171228 23343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:09.189780 23343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:09.193898 23343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:09.215994 23343 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.124:46245
I20260926 23:17:09.216051 23603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.124:46245 every 8 connection(s)
I20260926 23:17:09.217193 23343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
I20260926 23:17:09.224848 23604 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:17:09.228471 23604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Bootstrap starting.
I20260926 23:17:09.246336 23604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:17:09.246929 23604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Bootstrap complete.
I20260926 23:17:09.248025 23604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } }
I20260926 23:17:09.248332 23604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93301179318146f7a4aec1331bd3df3f, State: Initialized, Role: LEARNER
I20260926 23:17:09.248574 23604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } }
I20260926 23:17:09.249413 23607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.249742 23604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:09.249915 23607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: This master's current role is: LEARNER
I20260926 23:17:09.256749 23618 catalog_manager.cc:1259] Loaded cluster ID: cc985ffcca974c9cbabddd9fd97737ee
I20260926 23:17:09.256835 23618 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: loading cluster ID for follower catalog manager: success
I20260926 23:17:09.260100 23618 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: acquiring CA information for follower catalog manager: success
I20260926 23:17:09.262780 23618 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 23:17:09.499360 23541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:09.499696 23541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:09.499876 23541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:09.508577 23541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:09.508679 23541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:09.508800 23541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:17:09.519006 23541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab
--principal=oryx/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:35737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=43567
--tserver_master_addrs=127.16.62.126:38973,127.16.62.125:43067,127.16.62.124:46245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:09.520442 23541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:09.521752 23541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:17:09.526939 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.519081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.16.62.124:46245)
0926 23:17:09.520391 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:09.520395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:09.520412 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:17:09.520473 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:09.520476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:09.520615 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:09.520744 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:09.521426 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.522950 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.524871 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.525666 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.525823 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.526088 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:09.526429 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 23:17:09.526574 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":907,"threads_started":1}
I20260926 23:17:09.527060 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.518642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.0.0.1:44444)
0926 23:17:09.518916 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:09.518946 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:09.519102 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:09.520766 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:09.520771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:09.520799 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:09.521229 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.521241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.523128 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.523134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.524717 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.524730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.525867 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.525873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.526259 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.526285 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 23:17:09.526401 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 23:17:09.526578 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 23:17:09.528064 23579 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 23:17:09.528792 23607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 144eee699c8e4b32b55bb3f17e2e659d. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.529223 23607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: This master's current role is: LEARNER
W20260926 23:17:09.541019 23624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:09.541342 23627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:09.541635 23625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:09.545532 23541 server_base.cc:1034] running on GCE node
I20260926 23:17:09.546114 23541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:09.547034 23541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:09.548269 23541 hybrid_clock.cc:648] HybridClock initialized: now 1790464629548236 us; error 49 us; skew 500 ppm
I20260926 23:17:09.548585 23541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:09 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:09.551657 23541 init.cc:377] Logged in from keytab as oryx/127.16.62.65@KRBTEST.COM (short username oryx)
I20260926 23:17:09.553264 23541 webserver.cc:460] Webserver started at http://127.16.62.65:43567/ using document root <none> and password file <none>
I20260926 23:17:09.553974 23541 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:09.554078 23541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:09.559741 23541 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 23:17:09.562382 23634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:09.563720 23541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:17:09.563898 23541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "5aba1825270a44f091e6ed417461c3a3"
format_stamp: "Formatted at 2026-09-26 23:17:06 on dist-test-slave-vcv1"
server_key: "07f088eb47d89a79dd2b9d5008d7117c"
server_key_iv: "672f457700d9f7fbf6295e7a61b48ca6"
server_key_version: "encryptionkey@0"
I20260926 23:17:09.564932 23541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:09.613718 23541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:09.617708 23541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:09.617985 23541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:09.619074 23541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:09.620440 23541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:09.620556 23541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:09.620692 23541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:09.620783 23541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:09.674204 23541 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:35737
I20260926 23:17:09.674288 23746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:35737 every 8 connection(s)
I20260926 23:17:09.675611 23541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:17:09.685707 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 23541
I20260926 23:17:09.686008 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 22977
Sep 26 23:17:09 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.124@KRBTEST.COM
Sep 26 23:17:09 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.125@KRBTEST.COM
Sep 26 23:17:09 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464629, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:09.707085 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.678605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:58331 (local address 127.16.62.124:46245)
0926 23:17:09.679652 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:09.679656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:09.681375 (+ 1719us) server_negotiation.cc:407] Connection header received
0926 23:17:09.684176 (+ 2801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:09.684185 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:09.684293 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:09.684382 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:09.687252 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.688461 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.691720 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.692600 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.692731 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.698093 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:09.698099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:09.698104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:09.698155 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:09.701849 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.703758 (+ 1909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.703764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.703770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.703905 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.704425 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.704430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.704433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.704718 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:09.704839 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:09.706530 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0926 23:17:09.706756 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936}
Sep 26 23:17:09 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.66@KRBTEST.COM
I20260926 23:17:09.708592 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.679889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.16.62.65:58331)
0926 23:17:09.681275 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:09.681296 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:09.683919 (+ 2623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:09.684923 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:09.684939 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:09.686062 (+ 1123us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:09.687021 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.687049 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.688698 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.688705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.691399 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.691418 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.693114 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.693123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.693459 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.694871 (+ 1412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:09.694911 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:09.697857 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:09.702425 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:09.702436 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:09.702725 (+ 289us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:09.703590 (+ 865us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:09.704023 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:09.704029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:09.704036 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:09.704306 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:09.704932 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:09.704939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:09.705337 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 23:17:09.705643 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 23:17:09.706092 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":137,"thread_start_us":960,"threads_started":1}
I20260926 23:17:09.710596 23747 heartbeater.cc:347] Connected to a master server at 127.16.62.124:46245
I20260926 23:17:09.710893 23747 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:09.711737 23747 heartbeater.cc:511] Master 127.16.62.124:46245 requested a full tablet report, sending...
I20260926 23:17:09.711865 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.681381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.16.62.65:55437)
0926 23:17:09.681927 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:09.681945 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:09.691536 (+ 9591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:09.693496 (+ 1960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:09.693501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:09.694310 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:09.694636 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.694648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.698876 (+ 4228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.698884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.700526 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.700539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.701534 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.701540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.701759 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.702499 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:09.702522 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:09.705345 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:09.708996 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:09.709000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:09.709003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:09.709479 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:09.710113 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:09.710118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:09.710121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:09.710279 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:09.710896 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:09.710900 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:09.711053 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 23:17:09.711173 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 23:17:09.711414 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":21,"thread_start_us":572,"threads_started":1}
I20260926 23:17:09.712935 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.681779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:55437 (local address 127.16.62.125:43067)
0926 23:17:09.693099 (+ 11320us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:09.693104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:09.693127 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:17:09.693193 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:09.693198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:09.693280 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:09.697582 (+ 4302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:09.697772 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.698686 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.700700 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.701348 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.701480 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.705559 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:09.705564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:09.705568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:09.705604 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:09.708825 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.709780 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.709783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.709785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.709881 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.710427 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.710431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.710434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.710759 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:09.712177 (+ 1418us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:09.712290 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 23:17:09.712499 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11204,"thread_start_us":11066,"threads_started":1}
I20260926 23:17:09.713313 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.678625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:48621 (local address 127.16.62.126:38973)
0926 23:17:09.679104 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:09.679111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:09.687764 (+ 8653us) server_negotiation.cc:407] Connection header received
0926 23:17:09.687828 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:09.687833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:09.687905 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:09.688001 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:09.696291 (+ 8290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.697430 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.700979 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.701438 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.701556 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.706569 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:09.706574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:09.706585 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:09.706612 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:09.710210 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.710992 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.711000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.711003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.711152 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.711646 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.711652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.711655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.711987 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:09.712109 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:09.712745 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 23:17:09.712956 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260926 23:17:09.714131 23749 heartbeater.cc:347] Connected to a master server at 127.16.62.125:43067
I20260926 23:17:09.714265 23749 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:09.714484 23569 ts_manager.cc:194] Registered new tserver with Master: 5aba1825270a44f091e6ed417461c3a3 (127.16.62.65:35737)
I20260926 23:17:09.714599 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.679289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:48621)
0926 23:17:09.680307 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:09.680365 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:09.683679 (+ 3314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:09.692773 (+ 9094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:09.692780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:09.693277 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:09.694280 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.694292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.698543 (+ 4251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.698548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.700869 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:09.700880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.702059 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.702066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:09.702226 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.702814 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:09.702838 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:09.706397 (+ 3559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:09.710337 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:09.710341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:09.710362 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:09.710856 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:09.711308 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:09.711316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:09.711320 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:09.711491 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:09.712233 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:09.712238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:09.712427 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 23:17:09.713823 (+ 1396us) client_negotiation.cc:240] Negotiation successful
0926 23:17:09.714083 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":5,"thread_start_us":522,"threads_started":1}
I20260926 23:17:09.714709 23749 heartbeater.cc:511] Master 127.16.62.125:43067 requested a full tablet report, sending...
I20260926 23:17:09.715335 23748 heartbeater.cc:347] Connected to a master server at 127.16.62.126:38973
I20260926 23:17:09.715480 23748 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:09.715896 23748 heartbeater.cc:511] Master 127.16.62.126:38973 requested a full tablet report, sending...
I20260926 23:17:09.716928 22782 ts_manager.cc:194] Re-registered known tserver with Master: 5aba1825270a44f091e6ed417461c3a3 (127.16.62.65:35737)
I20260926 23:17:09.717139 23379 ts_manager.cc:194] Registered new tserver with Master: 5aba1825270a44f091e6ed417461c3a3 (127.16.62.65:35737)
I20260926 23:17:09.717499 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:09.686953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.16.62.66:42235)
0926 23:17:09.693759 (+ 6806us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:09.693769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:09.694613 (+ 844us) server_negotiation.cc:407] Connection header received
0926 23:17:09.694790 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:09.694796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:09.695031 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:09.695155 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:09.697546 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.699358 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.702005 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:09.703310 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:09.703427 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:09.710340 (+ 6913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:09.710370 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:09.710383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:09.710430 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:09.714072 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.714908 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.714915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.714918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.715066 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:09.715475 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:09.715480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:09.715483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:09.715923 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:09.716241 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:09.716779 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 23:17:09.717118 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6591,"thread_start_us":6777,"threads_started":1}
I20260926 23:17:09.717897 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:48621
I20260926 23:17:09.719369 23100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:09.805425 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 22977
I20260926 23:17:09.821887 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:42235
--local_ip_for_outbound_sockets=127.16.62.66
--tserver_master_addrs=127.16.62.126:38973
--webserver_port=34413
--webserver_interface=127.16.62.66
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab
--principal=oryx/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.62.126:38973,127.16.62.125:43067
--tserver_master_addrs=127.16.62.126:38973,127.16.62.125:43067,127.16.62.124:46245 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:17:09.972292 23527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: NON_VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: true } } }
I20260926 23:17:09.972832 23527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:09.974552 23761 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: attempting to promote NON_VOTER 93301179318146f7a4aec1331bd3df3f to VOTER
I20260926 23:17:09.975329 23761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } }
I20260926 23:17:09.977388 23579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 LEARNER]: Refusing update from remote peer 144eee699c8e4b32b55bb3f17e2e659d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 23:17:09.977579 23389 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Refusing update from remote peer 144eee699c8e4b32b55bb3f17e2e659d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 23:17:09.978067 23760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 23:17:09.978478 23527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 23:17:09.981523 23389 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 93301179318146f7a4aec1331bd3df3f (127.16.62.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:09.980886 23760 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 93301179318146f7a4aec1331bd3df3f (127.16.62.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:09.986331 23579 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 93301179318146f7a4aec1331bd3df3f (127.16.62.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:09.989822 23760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:09.989995 23527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:09.990257 23527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:09.990288 23760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:09.990473 23527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:09.990551 23607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:09.990715 23527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:09.990761 23607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:17:09.985911 23764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "144eee699c8e4b32b55bb3f17e2e659d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:09.991287 23764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: This master's current role is: FOLLOWER
W20260926 23:17:10.141271 23759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:10.141556 23759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:10.141687 23759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:10.148662 23759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:10.148751 23759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:10.148852 23759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:17:10.158275 23759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab
--principal=oryx/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:42235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=34413
--tserver_master_addrs=127.16.62.126:38973,127.16.62.125:43067,127.16.62.124:46245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:10.159435 23759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:10.160697 23759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:10.173743 23769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:10.173799 23770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:10.173815 23772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:10.177641 23759 server_base.cc:1034] running on GCE node
I20260926 23:17:10.178198 23759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:10.179083 23759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:10.180294 23759 hybrid_clock.cc:648] HybridClock initialized: now 1790464630180228 us; error 80 us; skew 500 ppm
I20260926 23:17:10.180593 23759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:10 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464630, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:10.183665 23759 init.cc:377] Logged in from keytab as oryx/127.16.62.66@KRBTEST.COM (short username oryx)
I20260926 23:17:10.185122 23759 webserver.cc:460] Webserver started at http://127.16.62.66:34413/ using document root <none> and password file <none>
I20260926 23:17:10.185719 23759 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:10.185839 23759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:10.191159 23759 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 23:17:10.193825 23779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:10.195118 23759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:17:10.195276 23759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "b0d13abca8344a939dc70777ed62c2f4"
format_stamp: "Formatted at 2026-09-26 23:17:07 on dist-test-slave-vcv1"
server_key: "026e8d6b63b356f261c4f998b82f5826"
server_key_iv: "7905d6c3ed1f4c3692402b0d9f555f37"
server_key_version: "encryptionkey@0"
I20260926 23:17:10.196244 23759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:10.214722 23759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:10.219586 23759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:10.219871 23759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:10.220932 23759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:10.222635 23759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:10.222733 23759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:10.222824 23759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:10.222890 23759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:10.256119 23759 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:42235
I20260926 23:17:10.256202 23891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:42235 every 8 connection(s)
I20260926 23:17:10.257541 23759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:17:10.260491 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 23759
I20260926 23:17:10.260802 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 23110
Sep 26 23:17:10 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464630, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
Sep 26 23:17:10 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.67@KRBTEST.COM
Sep 26 23:17:10 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464630, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.125@KRBTEST.COM
I20260926 23:17:10.296710 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.260572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:52725 (local address 127.16.62.126:38973)
0926 23:17:10.261114 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.261120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.273023 (+ 11903us) server_negotiation.cc:407] Connection header received
0926 23:17:10.276065 (+ 3042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.276074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.276224 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.276368 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.278153 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.279714 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.282207 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.283094 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.283252 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.289397 (+ 6145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.289403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.289407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.289446 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.292815 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.293742 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.293760 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.293766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.293924 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.294492 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.294498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.294501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.294870 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.295002 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.296056 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.296309 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":227,"threads_started":1}
Sep 26 23:17:10 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464630, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.124@KRBTEST.COM
I20260926 23:17:10.298712 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.262113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.16.62.67:41175)
0926 23:17:10.277130 (+ 15017us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.277142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.277176 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:17:10.277251 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.277261 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.277540 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.277737 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.279289 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.280611 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.282423 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.282921 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.283052 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.289447 (+ 6395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.289459 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.289475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.289513 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.294764 (+ 5251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.296068 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.296077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.296081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.296269 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.296723 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.296729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.296732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.297362 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.297659 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.298020 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.298290 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14800,"thread_start_us":14574,"threads_started":1}
I20260926 23:17:10.299100 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.261313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.66:52725)
0926 23:17:10.272369 (+ 11056us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:10.272419 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:10.275675 (+ 3256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:10.276698 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:10.276713 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:10.277298 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:10.277962 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.277989 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.279907 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.279912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.282050 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.282066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.283905 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.283914 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.284309 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.285059 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:10.285096 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:10.289107 (+ 4011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:10.292975 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.292985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.293001 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.293601 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.294086 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.294090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.294094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.294337 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.295086 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:10.295091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:10.295575 (+ 484us) client_negotiation.cc:769] Sending connection context
0926 23:17:10.297008 (+ 1433us) client_negotiation.cc:240] Negotiation successful
0926 23:17:10.297528 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10707,"thread_start_us":10472,"threads_started":1}
I20260926 23:17:10.300396 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.261950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.16.62.66:45539)
0926 23:17:10.280331 (+ 18381us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:10.280350 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:10.280521 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:10.281148 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:10.281155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:10.281940 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:10.283583 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.283596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.285065 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.285069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.286950 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.286964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.287956 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.287961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.288170 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.288899 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:10.288925 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:10.292873 (+ 3948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:10.297535 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.297539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.297543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.298138 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.298589 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.298593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.298597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.298736 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.299419 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:10.299423 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:10.299591 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 23:17:10.299712 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 23:17:10.299952 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18252,"thread_start_us":18013,"threads_started":1}
I20260926 23:17:10.300601 23893 heartbeater.cc:347] Connected to a master server at 127.16.62.126:38973
I20260926 23:17:10.300912 23893 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:10.301288 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.261406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:45539 (local address 127.16.62.125:43067)
0926 23:17:10.273406 (+ 12000us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.273416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.280425 (+ 7009us) server_negotiation.cc:407] Connection header received
0926 23:17:10.280672 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.280679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.280771 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.280854 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.283717 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.284893 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.287078 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.287729 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.287902 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.293073 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.293081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.293089 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.293138 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.297396 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.298327 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.298335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.298340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.298467 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.298921 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.298927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.298930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.299276 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.300632 (+ 1356us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.300749 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.300963 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11846,"thread_start_us":14675,"threads_started":1}
I20260926 23:17:10.301753 23893 heartbeater.cc:511] Master 127.16.62.126:38973 requested a full tablet report, sending...
I20260926 23:17:10.302526 23233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:10.303817 22782 ts_manager.cc:194] Re-registered known tserver with Master: b0d13abca8344a939dc70777ed62c2f4 (127.16.62.66:42235)
I20260926 23:17:10.304878 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:52725
I20260926 23:17:10.305440 23894 heartbeater.cc:347] Connected to a master server at 127.16.62.125:43067
I20260926 23:17:10.305591 23894 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:10.306078 23894 heartbeater.cc:511] Master 127.16.62.125:43067 requested a full tablet report, sending...
I20260926 23:17:10.306079 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.280688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:52441 (local address 127.16.62.124:46245)
0926 23:17:10.283579 (+ 2891us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.283586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.287526 (+ 3940us) server_negotiation.cc:407] Connection header received
0926 23:17:10.287796 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.287818 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.287940 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.288017 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.290532 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.291348 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.293282 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.293941 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.294047 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.298576 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.298582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.298586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.298634 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.302519 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.303277 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.303284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.303287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.303425 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.303872 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.303881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.303887 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.304383 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.304565 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.304971 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.305255 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2748,"thread_start_us":2619,"threads_started":1}
I20260926 23:17:10.307194 23379 ts_manager.cc:194] Registered new tserver with Master: b0d13abca8344a939dc70777ed62c2f4 (127.16.62.66:42235)
I20260926 23:17:10.308055 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.283530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.16.62.66:52441)
0926 23:17:10.287431 (+ 3901us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:10.287449 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:10.287625 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:10.288261 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:10.288267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:10.289019 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:10.290367 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.290379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.291468 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.291476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.293113 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.293129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.294364 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.294371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.294560 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.295122 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:10.295147 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:10.298377 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:10.302685 (+ 4308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.302690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.302695 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.303153 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.303566 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.303571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.303574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.303717 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.304526 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:10.304531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:10.304669 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 23:17:10.304802 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 23:17:10.305136 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3765,"thread_start_us":5812,"threads_started":1}
I20260926 23:17:10.309029 23892 heartbeater.cc:347] Connected to a master server at 127.16.62.124:46245
I20260926 23:17:10.309152 23892 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:10.309512 23892 heartbeater.cc:511] Master 127.16.62.124:46245 requested a full tablet report, sending...
I20260926 23:17:10.310485 23569 ts_manager.cc:194] Registered new tserver with Master: b0d13abca8344a939dc70777ed62c2f4 (127.16.62.66:42235)
I20260926 23:17:10.389701 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 23110
I20260926 23:17:10.406519 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:41175
--local_ip_for_outbound_sockets=127.16.62.67
--tserver_master_addrs=127.16.62.126:38973
--webserver_port=44439
--webserver_interface=127.16.62.67
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab
--principal=oryx/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.62.126:38973,127.16.62.125:43067
--tserver_master_addrs=127.16.62.126:38973,127.16.62.125:43067,127.16.62.124:46245 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:17:10.720074 23748 heartbeater.cc:503] Master 127.16.62.126:38973 was elected leader, sending a full tablet report...
W20260926 23:17:10.726907 23905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:10.727355 23905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:10.727541 23905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:10.734858 23905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:10.735046 23905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:10.735227 23905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:17:10.744930 23905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab
--principal=oryx/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:41175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=44439
--tserver_master_addrs=127.16.62.126:38973,127.16.62.125:43067,127.16.62.124:46245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:10.746230 23905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:10.747537 23905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:10.761101 23910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:10.761173 23913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:10.761140 23911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:10.762007 23905 server_base.cc:1034] running on GCE node
I20260926 23:17:10.762614 23905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:10.763487 23905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:10.764707 23905 hybrid_clock.cc:648] HybridClock initialized: now 1790464630764672 us; error 51 us; skew 500 ppm
I20260926 23:17:10.765030 23905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:10 dist-test-slave-vcv1 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464630, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:10.768011 23905 init.cc:377] Logged in from keytab as oryx/127.16.62.67@KRBTEST.COM (short username oryx)
I20260926 23:17:10.769902 23905 webserver.cc:460] Webserver started at http://127.16.62.67:44439/ using document root <none> and password file <none>
I20260926 23:17:10.770582 23905 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:10.770701 23905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:10.775873 23905 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:10.778292 23920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:10.779692 23905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:17:10.779870 23905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "dccd0d1f512249ebaed630f09bbc3c6f"
format_stamp: "Formatted at 2026-09-26 23:17:07 on dist-test-slave-vcv1"
server_key: "1aec6b4d49eacd8feb683dd642f7eee0"
server_key_iv: "6272af526ec3da29f1d884ffe8f4ecf3"
server_key_version: "encryptionkey@0"
I20260926 23:17:10.780982 23905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:10.797039 23905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:10.801226 23905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:10.801496 23905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:10.802418 23905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:10.803771 23905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:10.803869 23905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:10.803944 23905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:10.803995 23905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:10.834421 23905 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:41175
I20260926 23:17:10.834501 24032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:41175 every 8 connection(s)
I20260926 23:17:10.836012 23905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:17:10.845671 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 23905
Sep 26 23:17:10 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464630, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.124@KRBTEST.COM
Sep 26 23:17:10 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464630, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:10.885802 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.838785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:52005 (local address 127.16.62.124:46245)
0926 23:17:10.853913 (+ 15128us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.853917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.854564 (+ 647us) server_negotiation.cc:407] Connection header received
0926 23:17:10.860906 (+ 6342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.860913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.861016 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.861150 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.862272 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.863403 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.866735 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.867512 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.867644 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.876612 (+ 8968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.876621 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.876628 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.876676 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.881417 (+ 4741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.882427 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.882436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.882445 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.882612 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.883101 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.883106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.883110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.883482 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.883643 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.885152 (+ 1509us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.885410 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15039,"thread_start_us":15111,"threads_started":1}
I20260926 23:17:10.891640 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.840232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:47075 (local address 127.16.62.126:38973)
0926 23:17:10.850902 (+ 10670us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.850911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.850936 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:17:10.851010 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.851018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.851132 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.851266 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.862173 (+ 10907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.863401 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.872745 (+ 9344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.873620 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.873766 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.881466 (+ 7700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.881474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.881481 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.881523 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.885583 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.886460 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.886468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.886474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.886676 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.889513 (+ 2837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.889519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.889524 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.890050 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.890260 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.890817 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.891125 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10498,"thread_start_us":10394,"threads_started":1}
I20260926 23:17:10.891825 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.839650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.16.62.67:52005)
0926 23:17:10.854449 (+ 14799us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:10.854469 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:10.854596 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:10.861188 (+ 6592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:10.861196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:10.861844 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:10.862145 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.862157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.863522 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.863527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.866564 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.866585 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.868804 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.868814 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.869166 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.870159 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:10.870199 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:10.876372 (+ 6173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:10.881520 (+ 5148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.881532 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.881555 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.882309 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.882713 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.882720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.882726 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.882995 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.883594 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:10.883606 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:10.884618 (+ 1012us) client_negotiation.cc:769] Sending connection context
0926 23:17:10.885042 (+ 424us) client_negotiation.cc:240] Negotiation successful
0926 23:17:10.887251 (+ 2209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14667,"mutex_wait_us":73,"thread_start_us":14913,"threads_started":1}
I20260926 23:17:10.892711 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.839649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.67:47075)
0926 23:17:10.840549 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:10.840594 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:10.844278 (+ 3684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:10.851900 (+ 7622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:10.851918 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:10.854886 (+ 2968us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:10.861958 (+ 7072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.861988 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.863522 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.863527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.872549 (+ 9022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.872585 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.874127 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.874145 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.874397 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.876889 (+ 2492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:10.876917 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:10.881262 (+ 4345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:10.885690 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.885697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.885703 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.886315 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.889163 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.889171 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.889178 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.889381 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.890235 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:10.890243 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:10.890467 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 23:17:10.890632 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 23:17:10.891030 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":3,"thread_start_us":603,"threads_started":1}
I20260926 23:17:10.894356 24034 heartbeater.cc:347] Connected to a master server at 127.16.62.126:38973
I20260926 23:17:10.894717 24034 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:10.895711 24034 heartbeater.cc:511] Master 127.16.62.126:38973 requested a full tablet report, sending...
I20260926 23:17:10.897929 22782 ts_manager.cc:194] Re-registered known tserver with Master: dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175)
I20260926 23:17:10.899660 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:47075
Sep 26 23:17:10 dist-test-slave-vcv1 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464630, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.125@KRBTEST.COM
I20260926 23:17:10.910905 24033 heartbeater.cc:347] Connected to a master server at 127.16.62.124:46245
I20260926 23:17:10.911109 24033 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:10.911698 24033 heartbeater.cc:511] Master 127.16.62.124:46245 requested a full tablet report, sending...
I20260926 23:17:10.913280 23569 ts_manager.cc:194] Registered new tserver with Master: dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175)
I20260926 23:17:10.920019 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.855323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.16.62.67:41601)
0926 23:17:10.892387 (+ 37064us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:10.892411 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:10.892546 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:10.894818 (+ 2272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:10.894825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:10.895473 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:10.895747 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.895762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.897771 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.897780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.899655 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.899674 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.901556 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.901565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.901925 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.902908 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:10.902940 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:10.908080 (+ 5140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:10.915576 (+ 7496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.915582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.915586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.916439 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.917194 (+ 755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:10.917199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:10.917202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:10.917555 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:10.918598 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:10.918604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:10.918802 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 23:17:10.918938 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 23:17:10.919364 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36925,"thread_start_us":53899,"threads_started":1}
I20260926 23:17:10.921281 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.855447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:41601 (local address 127.16.62.125:43067)
0926 23:17:10.881324 (+ 25877us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.881333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.893396 (+ 12063us) server_negotiation.cc:407] Connection header received
0926 23:17:10.893604 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.893612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.894456 (+ 844us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.894602 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.895978 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.897538 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.899859 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.901052 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.901216 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.908411 (+ 7195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.908421 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.908429 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.908517 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.915258 (+ 6741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.916669 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.916674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.916678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.916948 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.917721 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.917728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.917734 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.918373 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.918516 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.920369 (+ 1853us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.920596 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25745,"thread_start_us":36151,"threads_started":1}
I20260926 23:17:10.922302 24035 heartbeater.cc:347] Connected to a master server at 127.16.62.125:43067
I20260926 23:17:10.922501 24035 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:10.923235 24035 heartbeater.cc:511] Master 127.16.62.125:43067 requested a full tablet report, sending...
I20260926 23:17:10.925602 23379 ts_manager.cc:194] Registered new tserver with Master: dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175)
I20260926 23:17:10.929330 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.854924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.16.62.126:38973)
0926 23:17:10.893800 (+ 38876us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.893808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.893830 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:17:10.893883 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.893890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.894001 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.894110 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.900462 (+ 6352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.901835 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.906904 (+ 5069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.907985 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.908101 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.916832 (+ 8731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.916839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.916846 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.916888 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.923365 (+ 6477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.924782 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.924790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.924796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.925010 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.926536 (+ 1526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.926541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.926555 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.927106 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.927457 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.928345 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.928711 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38146,"thread_start_us":48245,"threads_started":1}
I20260926 23:17:10.934213 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.860871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.16.62.125:43067)
0926 23:17:10.906221 (+ 45350us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.906226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.906244 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:17:10.906286 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.906290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.906388 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.906497 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.911909 (+ 5412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.918343 (+ 6434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.921681 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.922878 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.923100 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.927846 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.927852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.927857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.927919 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.931316 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.932077 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.932081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.932083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.932274 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.932728 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.932733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.932737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.932989 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.933253 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.933569 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.933773 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29705,"thread_start_us":26372,"threads_started":1}
I20260926 23:17:10.936471 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.887916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.16.62.124:46245)
0926 23:17:10.888109 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.888116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.904606 (+ 16490us) server_negotiation.cc:407] Connection header received
0926 23:17:10.904911 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.904917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.905038 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.905137 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:10.916181 (+ 11044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.917620 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.920566 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.921545 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.921637 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.928666 (+ 7029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:10.928673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:10.928677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:10.928713 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:10.932530 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.934294 (+ 1764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.934299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.934302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.934554 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:10.935065 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:10.935070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:10.935073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:10.935424 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:10.935642 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.936033 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.936261 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:17:10.934515 22782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:17:10.942361 22782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:17:10.981004 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.961418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.16.62.65:35737)
0926 23:17:10.974859 (+ 13441us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.974865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.974884 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:17:10.974948 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.974952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.975093 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.975219 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:10.975691 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.976873 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.978981 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.979927 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.980073 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.980328 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.980563 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.980712 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13278,"thread_start_us":13489,"threads_started":1}
I20260926 23:17:10.981024 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.961203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35737 (local address 127.0.0.1:47234)
0926 23:17:10.972130 (+ 10927us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:10.972172 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:10.972288 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:10.975243 (+ 2955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:10.975251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:10.975279 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:10.975544 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.975551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.977015 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.977023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.978834 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.978845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.980163 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.980170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.980369 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.980404 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:17:10.980523 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 23:17:10.980655 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10788,"thread_start_us":9204,"threads_started":1}
I20260926 23:17:10.982393 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.960344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41175 (local address 127.0.0.1:53850)
0926 23:17:10.968973 (+ 8629us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:10.968993 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:10.969107 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:10.974187 (+ 5080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:10.974191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:10.974212 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:10.974611 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.974625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.978256 (+ 3631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.978259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.979820 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.979831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.981642 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.981648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.981826 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.981861 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:17:10.981947 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:17:10.982068 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":280,"threads_started":1}
I20260926 23:17:10.982434 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.960577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.16.62.67:41175)
0926 23:17:10.973698 (+ 13121us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.973705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.973739 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:17:10.973833 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.973838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.974021 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.974209 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:10.975478 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.977058 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.979969 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.981473 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.981600 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.981837 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.981992 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.982131 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12961,"thread_start_us":12814,"threads_started":1}
I20260926 23:17:10.984959 23682 tablet_service.cc:1467] Processing CreateTablet for tablet 970a03f392d54f1092f7a64a9845129a (DEFAULT_TABLE table=test-table [id=6651a417736646a98b971bb0525c57c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:10.986130 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.960933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:42235 (local address 127.0.0.1:50828)
0926 23:17:10.970745 (+ 9812us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:10.970768 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:10.970916 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:10.974738 (+ 3822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:10.974742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:10.974765 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:10.975106 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.975118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.979999 (+ 4881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.980005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.983379 (+ 3374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:10.983395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.985296 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.985302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:10.985541 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.985573 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:17:10.985659 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:17:10.985773 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9645,"thread_start_us":10930,"threads_started":1}
I20260926 23:17:10.986393 23682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970a03f392d54f1092f7a64a9845129a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:10.985932 23968 tablet_service.cc:1467] Processing CreateTablet for tablet 970a03f392d54f1092f7a64a9845129a (DEFAULT_TABLE table=test-table [id=6651a417736646a98b971bb0525c57c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:10.986835 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:10.961361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.16.62.66:42235)
0926 23:17:10.974212 (+ 12851us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:10.974220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:10.974246 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:17:10.974373 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:10.974381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:10.974562 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:10.974753 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:10.975325 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.976903 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.983531 (+ 6628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:10.985089 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:10.985214 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:10.985376 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:10.986423 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0926 23:17:10.986570 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12727,"thread_start_us":12632,"threads_started":1}
I20260926 23:17:10.987269 23968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970a03f392d54f1092f7a64a9845129a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:10.990083 23827 tablet_service.cc:1467] Processing CreateTablet for tablet 970a03f392d54f1092f7a64a9845129a (DEFAULT_TABLE table=test-table [id=6651a417736646a98b971bb0525c57c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:10.991291 23827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970a03f392d54f1092f7a64a9845129a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:10.998531 24065 tablet_bootstrap.cc:492] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: Bootstrap starting.
I20260926 23:17:11.001374 24066 tablet_bootstrap.cc:492] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: Bootstrap starting.
I20260926 23:17:11.003053 24065 tablet_bootstrap.cc:654] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:11.004292 24065 log.cc:824] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:11.004699 24066 tablet_bootstrap.cc:654] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:11.006054 24068 tablet_bootstrap.cc:492] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: Bootstrap starting.
I20260926 23:17:11.006131 24066 log.cc:824] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:11.009241 24066 tablet_bootstrap.cc:492] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: No bootstrap required, opened a new log
I20260926 23:17:11.009335 24068 tablet_bootstrap.cc:654] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:11.009517 24066 ts_tablet_manager.cc:1397] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:17:11.009720 24065 tablet_bootstrap.cc:492] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: No bootstrap required, opened a new log
I20260926 23:17:11.009967 24065 ts_tablet_manager.cc:1397] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260926 23:17:11.010489 24068 log.cc:824] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:11.013242 24068 tablet_bootstrap.cc:492] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: No bootstrap required, opened a new log
I20260926 23:17:11.013480 24068 ts_tablet_manager.cc:1397] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:17:11.017735 24066 raft_consensus.cc:348] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } }
I20260926 23:17:11.018087 24066 raft_consensus.cc:374] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:11.018240 24066 raft_consensus.cc:729] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aba1825270a44f091e6ed417461c3a3, State: Initialized, Role: FOLLOWER
I20260926 23:17:11.018817 24066 consensus_queue.cc:260] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } }
I20260926 23:17:11.020387 24066 ts_tablet_manager.cc:1428] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260926 23:17:11.020188 24065 raft_consensus.cc:348] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } }
I20260926 23:17:11.020550 24065 raft_consensus.cc:374] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:11.020644 24065 raft_consensus.cc:729] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccd0d1f512249ebaed630f09bbc3c6f, State: Initialized, Role: FOLLOWER
I20260926 23:17:11.021266 24065 consensus_queue.cc:260] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } }
I20260926 23:17:11.025825 24034 heartbeater.cc:503] Master 127.16.62.126:38973 was elected leader, sending a full tablet report...
I20260926 23:17:11.026510 24065 ts_tablet_manager.cc:1428] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: Time spent starting tablet: real 0.016s user 0.011s sys 0.007s
I20260926 23:17:11.024844 24068 raft_consensus.cc:348] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } }
I20260926 23:17:11.027233 24068 raft_consensus.cc:374] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:11.027359 24068 raft_consensus.cc:729] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d13abca8344a939dc70777ed62c2f4, State: Initialized, Role: FOLLOWER
I20260926 23:17:11.027913 24068 consensus_queue.cc:260] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } }
I20260926 23:17:11.029871 23893 heartbeater.cc:503] Master 127.16.62.126:38973 was elected leader, sending a full tablet report...
I20260926 23:17:11.030651 24068 ts_tablet_manager.cc:1428] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260926 23:17:11.089764 24036 tablet.cc:2369] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:17:11.153193 24073 raft_consensus.cc:482] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:11.153537 24073 raft_consensus.cc:504] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } }
I20260926 23:17:11.155332 24073 leader_election.cc:288] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175), 5aba1825270a44f091e6ed417461c3a3 (127.16.62.65:35737)
I20260926 23:17:11.163563 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:11.155628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:58331 (local address 127.16.62.67:41175)
0926 23:17:11.155876 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:11.155883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:11.156907 (+ 1024us) server_negotiation.cc:407] Connection header received
0926 23:17:11.157108 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:11.157112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:11.157173 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:11.157238 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:11.157748 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.158698 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.161294 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.162510 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.162676 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:11.162737 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:11.163161 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 23:17:11.163289 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:17:11.163672 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:11.155961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:51649 (local address 127.16.62.65:35737)
0926 23:17:11.156221 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:11.156229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:11.156522 (+ 293us) server_negotiation.cc:407] Connection header received
0926 23:17:11.156761 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:11.156768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:11.156840 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:11.156945 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:11.157506 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.158510 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.161294 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.162513 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.162663 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:11.162741 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:11.163161 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 23:17:11.163285 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:17:11.164539 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:11.155624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41175 (local address 127.16.62.66:58331)
0926 23:17:11.156774 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:11.156797 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:11.156970 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:11.157285 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:11.157291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:11.157317 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:11.157602 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:11.157609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.158814 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.158821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:11.161137 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:11.161147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.162732 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.162741 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:11.162946 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:11.163044 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 23:17:11.163120 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 23:17:11.163302 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":197,"thread_start_us":515,"threads_started":1}
I20260926 23:17:11.164553 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:11.155591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35737 (local address 127.16.62.66:51649)
0926 23:17:11.156419 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:11.156433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:11.156602 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:11.156982 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:11.156986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:11.157011 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:11.157349 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:11.157363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.158659 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.158663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:11.161132 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:11.161147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.162768 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.162773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:11.162952 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:11.163020 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 23:17:11.163120 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 23:17:11.163302 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":132,"thread_start_us":865,"threads_started":1}
I20260926 23:17:11.165717 23702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970a03f392d54f1092f7a64a9845129a" candidate_uuid: "b0d13abca8344a939dc70777ed62c2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aba1825270a44f091e6ed417461c3a3" is_pre_election: true
I20260926 23:17:11.165892 23988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970a03f392d54f1092f7a64a9845129a" candidate_uuid: "b0d13abca8344a939dc70777ed62c2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" is_pre_election: true
I20260926 23:17:11.166430 23702 raft_consensus.cc:2454] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0d13abca8344a939dc70777ed62c2f4 in term 0.
I20260926 23:17:11.166771 23988 raft_consensus.cc:2454] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0d13abca8344a939dc70777ed62c2f4 in term 0.
I20260926 23:17:11.167389 23783 leader_election.cc:302] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aba1825270a44f091e6ed417461c3a3, b0d13abca8344a939dc70777ed62c2f4; no voters:
I20260926 23:17:11.167989 24073 raft_consensus.cc:2781] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:17:11.168210 24073 raft_consensus.cc:482] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:17:11.168365 24073 raft_consensus.cc:3037] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:11.170658 24073 raft_consensus.cc:504] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } }
I20260926 23:17:11.171240 24073 leader_election.cc:288] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [CANDIDATE]: Term 1 election: Requested vote from peers dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175), 5aba1825270a44f091e6ed417461c3a3 (127.16.62.65:35737)
I20260926 23:17:11.171695 23702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970a03f392d54f1092f7a64a9845129a" candidate_uuid: "b0d13abca8344a939dc70777ed62c2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aba1825270a44f091e6ed417461c3a3"
I20260926 23:17:11.171849 23988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970a03f392d54f1092f7a64a9845129a" candidate_uuid: "b0d13abca8344a939dc70777ed62c2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccd0d1f512249ebaed630f09bbc3c6f"
I20260926 23:17:11.171962 23702 raft_consensus.cc:3037] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:11.172115 23988 raft_consensus.cc:3037] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:11.177398 23988 raft_consensus.cc:2454] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d13abca8344a939dc70777ed62c2f4 in term 1.
I20260926 23:17:11.177398 23702 raft_consensus.cc:2454] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d13abca8344a939dc70777ed62c2f4 in term 1.
I20260926 23:17:11.177872 23783 leader_election.cc:302] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aba1825270a44f091e6ed417461c3a3, b0d13abca8344a939dc70777ed62c2f4; no voters:
I20260926 23:17:11.178257 24073 raft_consensus.cc:2781] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:11.178990 24073 raft_consensus.cc:686] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 1 LEADER]: Becoming Leader. State: Replica: b0d13abca8344a939dc70777ed62c2f4, State: Running, Role: LEADER
I20260926 23:17:11.179750 24073 consensus_queue.cc:237] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } }
W20260926 23:17:11.184896 23750 tablet.cc:2369] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:17:11.185894 22781 catalog_manager.cc:5495] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0d13abca8344a939dc70777ed62c2f4 (127.16.62.66). New cstate: current_term: 1 leader_uuid: "b0d13abca8344a939dc70777ed62c2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } health_report { overall_health: UNKNOWN } } }
I20260926 23:17:11.211885 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:11.206613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.16.62.66:42235)
0926 23:17:11.206835 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:11.206841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:11.207001 (+ 160us) server_negotiation.cc:407] Connection header received
0926 23:17:11.207217 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:11.207220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:11.207293 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:11.207425 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:11.207815 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.208472 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.209857 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.210255 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.210380 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:11.210842 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:11.211327 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:11.211475 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:11.211550 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 23:17:11.211625 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 23:17:11.220846 23702 raft_consensus.cc:1261] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [term 1 FOLLOWER]: Refusing update from remote peer b0d13abca8344a939dc70777ed62c2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:11.220919 23988 raft_consensus.cc:1261] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [term 1 FOLLOWER]: Refusing update from remote peer b0d13abca8344a939dc70777ed62c2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:11.221783 24076 consensus_queue.cc:1035] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:11.222152 24073 consensus_queue.cc:1035] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:11.235129 24080 mvcc.cc:205] Tried to move back new op lower bound from 7333743129471954944 to 7333743129317363712. Current Snapshot: MvccSnapshot[applied={T|T < 7333743129471954944}]
I20260926 23:17:11.251199 22781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:11.251464 22781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:11.258602 22781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 144eee699c8e4b32b55bb3f17e2e659d: Sending DeleteTablet for 3 replicas of tablet 970a03f392d54f1092f7a64a9845129a
I20260926 23:17:11.259615 23968 tablet_service.cc:1514] Processing DeleteTablet for tablet 970a03f392d54f1092f7a64a9845129a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:11 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:53850
I20260926 23:17:11.259750 23682 tablet_service.cc:1514] Processing DeleteTablet for tablet 970a03f392d54f1092f7a64a9845129a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:11 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:47234
I20260926 23:17:11.259754 23827 tablet_service.cc:1514] Processing DeleteTablet for tablet 970a03f392d54f1092f7a64a9845129a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:11 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:50828
I20260926 23:17:11.271934 24091 tablet_replica.cc:330] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: stopping tablet replica
I20260926 23:17:11.273118 24091 raft_consensus.cc:2229] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:17:11.274662 24091 raft_consensus.cc:2258] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:11.278349 24091 ts_tablet_manager.cc:1905] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:11.278640 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 22750
I20260926 23:17:11.279706 22812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:11.285346 24090 tablet_replica.cc:330] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: stopping tablet replica
I20260926 23:17:11.285416 24089 tablet_replica.cc:330] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: stopping tablet replica
I20260926 23:17:11.285799 24089 raft_consensus.cc:2229] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:11.286266 24089 raft_consensus.cc:2258] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:11.286247 24090 raft_consensus.cc:2229] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:11.286896 24090 raft_consensus.cc:2258] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:11.294734 24090 ts_tablet_manager.cc:1905] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:11.295974 24089 ts_tablet_manager.cc:1905] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:11.301913 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:11.265249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.16.62.125:43067)
0926 23:17:11.265454 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:11.265463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:11.277143 (+ 11680us) server_negotiation.cc:407] Connection header received
0926 23:17:11.279702 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:11.279708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:11.279790 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:11.279861 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:11.282139 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.283225 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.286623 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.287297 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.287383 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:11.291028 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:11.291046 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:11.291051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:11.291091 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:11.294667 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:11.299559 (+ 4892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:11.299564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:11.299568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:11.299709 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:11.300186 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:11.300192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:11.300197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:11.300539 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:11.300770 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:11.301358 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 23:17:11.301584 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 23:17:11.302539 24091 ts_tablet_manager.cc:1918] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:11.304277 24091 log.cc:1196] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/970a03f392d54f1092f7a64a9845129a
I20260926 23:17:11.304987 24091 ts_tablet_manager.cc:1939] T 970a03f392d54f1092f7a64a9845129a P b0d13abca8344a939dc70777ed62c2f4: Deleting consensus metadata
I20260926 23:17:11.308343 24090 ts_tablet_manager.cc:1918] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:11.308656 24090 log.cc:1196] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/970a03f392d54f1092f7a64a9845129a
I20260926 23:17:11.309237 24090 ts_tablet_manager.cc:1939] T 970a03f392d54f1092f7a64a9845129a P 5aba1825270a44f091e6ed417461c3a3: Deleting consensus metadata
I20260926 23:17:11.312557 24089 ts_tablet_manager.cc:1918] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:11.312779 24089 log.cc:1196] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/970a03f392d54f1092f7a64a9845129a
I20260926 23:17:11.313329 24089 ts_tablet_manager.cc:1939] T 970a03f392d54f1092f7a64a9845129a P dccd0d1f512249ebaed630f09bbc3c6f: Deleting consensus metadata
I20260926 23:17:11.325220 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:11.277604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.16.62.124:46245)
0926 23:17:11.277763 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:11.277768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:11.305410 (+ 27642us) server_negotiation.cc:407] Connection header received
0926 23:17:11.305647 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:11.305653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:11.305727 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:11.305796 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:11.310865 (+ 5069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.311955 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.313907 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.314564 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.314683 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:11.318495 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:11.318502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:11.318506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:11.318555 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:11.322019 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:11.322916 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:11.322922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:11.322925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:11.323081 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:11.323500 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:11.323505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:11.323509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:11.323853 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:11.324187 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:11.324469 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 23:17:11.324680 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:17:11.425123 22769 catalog_manager.cc:4841] TS 5aba1825270a44f091e6ed417461c3a3 (127.16.62.65:35737): tablet 970a03f392d54f1092f7a64a9845129a (table test-table [id=6651a417736646a98b971bb0525c57c4]) successfully deleted
I20260926 23:17:11.428783 22766 catalog_manager.cc:4841] TS dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175): tablet 970a03f392d54f1092f7a64a9845129a (table test-table [id=6651a417736646a98b971bb0525c57c4]) successfully deleted
I20260926 23:17:11.429044 22766 catalog_manager.cc:4841] TS b0d13abca8344a939dc70777ed62c2f4 (127.16.62.66:42235): tablet 970a03f392d54f1092f7a64a9845129a (table test-table [id=6651a417736646a98b971bb0525c57c4]) successfully deleted
I20260926 23:17:11.429431 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 22750
I20260926 23:17:11.501207 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:11.468348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.16.62.125:43067)
0926 23:17:11.468578 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:11.468587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:11.484847 (+ 16260us) server_negotiation.cc:407] Connection header received
0926 23:17:11.485129 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:11.485134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:11.485208 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:11.485297 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:11.487492 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.488724 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.490548 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.491027 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.491164 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:11.494338 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:11.494343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:11.494346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:11.494382 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:11.498251 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:11.499095 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:11.499102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:11.499105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:11.499242 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:11.499681 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:11.499698 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:11.499701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:11.500062 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:11.500444 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:11.500633 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 23:17:11.500853 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:17:11.501971 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:11.484963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.16.62.124:46245)
0926 23:17:11.485154 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:11.485160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:11.485754 (+ 594us) server_negotiation.cc:407] Connection header received
0926 23:17:11.486000 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:11.486005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:11.486080 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:11.486149 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:11.488186 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.489281 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.491410 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:11.492042 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:11.492203 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:11.495461 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:11.495468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:11.495472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:11.495512 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:11.498813 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:11.499700 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:11.499705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:11.499708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:11.499849 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:11.500325 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:11.500332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:11.500335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:11.500765 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:11.500965 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:11.501520 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 23:17:11.501746 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:17:12.244210 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.243072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:51985)
0926 23:17:12.243848 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.243988 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":319,"threads_started":1}
I20260926 23:17:12.244280 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.243367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.67:36671)
0926 23:17:12.243885 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.244028 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":297,"threads_started":1}
W20260926 23:17:12.244684 24034 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:38973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
W20260926 23:17:12.244792 23748 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:38973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
I20260926 23:17:12.245862 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.245173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.66:43917)
0926 23:17:12.245622 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.245713 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":227,"threads_started":1}
W20260926 23:17:12.246276 23893 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:38973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
I20260926 23:17:12.758524 24104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 144eee699c8e4b32b55bb3f17e2e659d)
I20260926 23:17:12.758564 24103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 144eee699c8e4b32b55bb3f17e2e659d)
I20260926 23:17:12.758838 24103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } }
I20260926 23:17:12.758821 24104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } }
I20260926 23:17:12.760425 24103 leader_election.cc:288] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973), 93301179318146f7a4aec1331bd3df3f (127.16.62.124:46245)
I20260926 23:17:12.766463 24104 leader_election.cc:288] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973), 0610f614ebd44277a007bea9a16e8d7b (127.16.62.125:43067)
I20260926 23:17:12.770056 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.760532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44344)
0926 23:17:12.769043 (+ 8511us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.769781 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8357,"thread_start_us":8540,"threads_started":1}
I20260926 23:17:12.770094 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.760615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44334)
0926 23:17:12.769536 (+ 8921us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.769800 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8754,"threads_started":1}
W20260926 23:17:12.770308 23364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
W20260926 23:17:12.770504 23554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
I20260926 23:17:12.773092 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.771646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44350)
0926 23:17:12.771863 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.771972 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260926 23:17:12.773378 23364 leader_election.cc:334] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973): Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
I20260926 23:17:12.776064 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.775668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44366)
0926 23:17:12.775849 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.775932 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:17:12.776314 23554 leader_election.cc:334] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973): Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
I20260926 23:17:12.779227 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.760563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:46245 (local address 127.0.0.1:34586)
0926 23:17:12.769307 (+ 8744us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.769330 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:12.769540 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:12.771612 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:12.771619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:12.772668 (+ 1049us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:12.773531 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:12.773547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:12.775421 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:12.775428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:12.777498 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:12.777514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:12.778752 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:12.778756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:12.778857 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:12.778881 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 23:17:12.778948 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 23:17:12.779020 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8632,"thread_start_us":8188,"threads_started":1}
I20260926 23:17:12.779567 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.769994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.16.62.124:46245)
0926 23:17:12.771036 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:12.771044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:12.771063 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:17:12.771135 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:12.771140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:12.771235 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:12.771342 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:12.773810 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:12.775288 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:12.777637 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:12.778588 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:12.778705 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:12.778803 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:12.779009 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 23:17:12.779126 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":786,"threads_started":1}
I20260926 23:17:12.780256 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.761616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:43067 (local address 127.0.0.1:33086)
0926 23:17:12.770111 (+ 8495us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.770130 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:12.771345 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:12.771968 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:12.771975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:12.772014 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:12.774679 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:12.774693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:12.776269 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:12.776274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:12.778000 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:12.778010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:12.779525 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:12.779530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:12.779725 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:12.779752 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 23:17:12.779843 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 23:17:12.779918 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8355,"thread_start_us":8153,"threads_started":1}
I20260926 23:17:12.780192 23579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0610f614ebd44277a007bea9a16e8d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "93301179318146f7a4aec1331bd3df3f" is_pre_election: true
I20260926 23:17:12.780344 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.761986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.16.62.125:43067)
0926 23:17:12.769607 (+ 7621us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:12.769614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:12.771490 (+ 1876us) server_negotiation.cc:407] Connection header received
0926 23:17:12.771565 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:12.771572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:12.771655 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:12.771737 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:12.774316 (+ 2579us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x61200000f228. stack: 00007faea82e066e 00007faea82e0430 00007faea73675ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007faea82d60fd 00007faea82d5d1e 00007faebb1df3cd 00007faebb1df36c 00007faeaee0fe6b 00007faeaedfc475 00007faeaed6aea5 00007faeaed69eff 00007faeaed9d82f 00007faeaed9d58d
0926 23:17:12.774908 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:12.776095 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:12.778133 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:12.779237 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:12.779348 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:12.779418 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:12.779882 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 23:17:12.780004 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":7498,"spinlock_wait_cycles":2874240,"thread_start_us":7341,"threads_started":1}
I20260926 23:17:12.780530 23579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0610f614ebd44277a007bea9a16e8d7b in term 1.
I20260926 23:17:12.780836 23389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93301179318146f7a4aec1331bd3df3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0610f614ebd44277a007bea9a16e8d7b" is_pre_election: true
I20260926 23:17:12.780974 23366 leader_election.cc:302] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0610f614ebd44277a007bea9a16e8d7b, 93301179318146f7a4aec1331bd3df3f; no voters: 144eee699c8e4b32b55bb3f17e2e659d
I20260926 23:17:12.781172 23389 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93301179318146f7a4aec1331bd3df3f in term 1.
I20260926 23:17:12.781358 24103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 23:17:12.781474 24103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 144eee699c8e4b32b55bb3f17e2e659d)
I20260926 23:17:12.781574 24103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:17:12.781633 23556 leader_election.cc:302] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0610f614ebd44277a007bea9a16e8d7b, 93301179318146f7a4aec1331bd3df3f; no voters: 144eee699c8e4b32b55bb3f17e2e659d
I20260926 23:17:12.781985 24104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 23:17:12.782094 24104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 144eee699c8e4b32b55bb3f17e2e659d)
I20260926 23:17:12.782183 24104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:17:12.784672 24103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } }
I20260926 23:17:12.785097 24104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } }
I20260926 23:17:12.785656 24104 leader_election.cc:288] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 2 election: Requested vote from peers 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973), 0610f614ebd44277a007bea9a16e8d7b (127.16.62.125:43067)
I20260926 23:17:12.785897 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.785386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44378)
0926 23:17:12.785598 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.785728 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 23:17:12.785892 23579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0610f614ebd44277a007bea9a16e8d7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "93301179318146f7a4aec1331bd3df3f"
I20260926 23:17:12.785956 24103 leader_election.cc:288] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [CANDIDATE]: Term 2 election: Requested vote from peers 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973), 93301179318146f7a4aec1331bd3df3f (127.16.62.124:46245)
I20260926 23:17:12.786222 23579 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0610f614ebd44277a007bea9a16e8d7b in current term 2: Already voted for candidate 93301179318146f7a4aec1331bd3df3f in this term.
I20260926 23:17:12.786681 23389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93301179318146f7a4aec1331bd3df3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0610f614ebd44277a007bea9a16e8d7b"
I20260926 23:17:12.786974 23389 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93301179318146f7a4aec1331bd3df3f in current term 2: Already voted for candidate 0610f614ebd44277a007bea9a16e8d7b in this term.
I20260926 23:17:12.787242 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.786799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44396)
0926 23:17:12.787006 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.787089 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 23:17:12.787607 23364 leader_election.cc:334] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973): Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
I20260926 23:17:12.787750 23366 leader_election.cc:302] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0610f614ebd44277a007bea9a16e8d7b; no voters: 144eee699c8e4b32b55bb3f17e2e659d, 93301179318146f7a4aec1331bd3df3f
I20260926 23:17:12.788034 24103 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 23:17:12.788453 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.786517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44384)
0926 23:17:12.788222 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.788331 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1588}
I20260926 23:17:12.789295 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:12.789014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44402)
0926 23:17:12.789132 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:12.789188 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:17:12.789513 23554 leader_election.cc:334] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973): Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
I20260926 23:17:12.789664 23554 leader_election.cc:302] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93301179318146f7a4aec1331bd3df3f; no voters: 0610f614ebd44277a007bea9a16e8d7b, 144eee699c8e4b32b55bb3f17e2e659d
I20260926 23:17:12.789952 24104 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 23:17:13.246145 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:13.245394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.67:41049)
0926 23:17:13.245885 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:13.245980 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":261,"threads_started":1}
I20260926 23:17:13.247092 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:13.245390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:36673)
0926 23:17:13.246781 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:13.246866 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1286,"thread_start_us":1026,"threads_started":1}
I20260926 23:17:13.248205 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:13.246847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.66:54861)
0926 23:17:13.247889 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:13.247982 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":859,"threads_started":1}
I20260926 23:17:14.247774 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.247077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.67:39621)
0926 23:17:14.247537 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.247618 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260926 23:17:14.250137 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.249083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.66:52237)
0926 23:17:14.249807 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.249901 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":504,"threads_started":1}
I20260926 23:17:14.250284 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.248721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:54255)
0926 23:17:14.249431 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.250057 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":467,"threads_started":1}
I20260926 23:17:14.314459 24119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:14.314714 24119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } }
I20260926 23:17:14.315404 24119 leader_election.cc:288] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973), 0610f614ebd44277a007bea9a16e8d7b (127.16.62.125:43067)
I20260926 23:17:14.315981 23389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93301179318146f7a4aec1331bd3df3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0610f614ebd44277a007bea9a16e8d7b" is_pre_election: true
I20260926 23:17:14.316259 23389 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93301179318146f7a4aec1331bd3df3f in term 2.
I20260926 23:17:14.316416 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.315542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44628)
0926 23:17:14.316070 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.316207 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":302,"threads_started":1}
I20260926 23:17:14.316845 23556 leader_election.cc:302] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0610f614ebd44277a007bea9a16e8d7b, 93301179318146f7a4aec1331bd3df3f; no voters:
I20260926 23:17:14.317155 24119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 23:17:14.317297 24119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:17:14.317399 24119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 2 FOLLOWER]: Advancing to term 3
I20260926 23:17:14.317910 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.317526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44634)
0926 23:17:14.317713 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.317800 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 23:17:14.318135 23554 leader_election.cc:334] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973): Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
I20260926 23:17:14.319583 24119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } }
I20260926 23:17:14.320109 24119 leader_election.cc:288] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 3 election: Requested vote from peers 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973), 0610f614ebd44277a007bea9a16e8d7b (127.16.62.125:43067)
I20260926 23:17:14.320607 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.320246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44638)
0926 23:17:14.320389 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.320465 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 23:17:14.320578 23389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93301179318146f7a4aec1331bd3df3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0610f614ebd44277a007bea9a16e8d7b"
I20260926 23:17:14.320785 23389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 2 FOLLOWER]: Advancing to term 3
I20260926 23:17:14.321400 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.321115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44646)
0926 23:17:14.321245 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.321312 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 23:17:14.321564 23554 leader_election.cc:334] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973): Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
I20260926 23:17:14.322935 23389 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93301179318146f7a4aec1331bd3df3f in term 3.
I20260926 23:17:14.323341 23556 leader_election.cc:302] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0610f614ebd44277a007bea9a16e8d7b, 93301179318146f7a4aec1331bd3df3f; no voters: 144eee699c8e4b32b55bb3f17e2e659d
I20260926 23:17:14.323618 24119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 3 FOLLOWER]: Leader election won for term 3
I20260926 23:17:14.323987 24119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [term 3 LEADER]: Becoming Leader. State: Replica: 93301179318146f7a4aec1331bd3df3f, State: Running, Role: LEADER
I20260926 23:17:14.324376 24119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } }
I20260926 23:17:14.326581 24122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93301179318146f7a4aec1331bd3df3f. Latest consensus state: current_term: 3 leader_uuid: "93301179318146f7a4aec1331bd3df3f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:14.326799 24122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:14.327383 24124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:17:14.330883 24124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:17:14.331665 24124 catalog_manager.cc:1259] Loaded cluster ID: cc985ffcca974c9cbabddd9fd97737ee
I20260926 23:17:14.331764 24124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:17:14.332971 24124 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:17:14.333722 24124 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Loaded TSK: 0
I20260926 23:17:14.334501 24124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:17:14.375412 23568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:17:14.382242 23568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:17:14.386683 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.386246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44668)
0926 23:17:14.386438 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.386540 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 23:17:14.386897 23389 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [term 3 FOLLOWER]: Refusing update from remote peer 93301179318146f7a4aec1331bd3df3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 23:17:14.387794 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.387360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44682)
0926 23:17:14.387565 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.387630 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 23:17:14.387820 24122 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 23:17:14.388155 23554 consensus_peers.cc:487] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f -> Peer 144eee699c8e4b32b55bb3f17e2e659d (127.16.62.126:38973): Couldn't send request to peer 144eee699c8e4b32b55bb3f17e2e659d. Status: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 23:17:14.391142 24126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93301179318146f7a4aec1331bd3df3f. Latest consensus state: current_term: 3 leader_uuid: "93301179318146f7a4aec1331bd3df3f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:14.391535 24126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:17:14.392866 24122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "93301179318146f7a4aec1331bd3df3f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:14.393215 24122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:14.393617 24126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "93301179318146f7a4aec1331bd3df3f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "144eee699c8e4b32b55bb3f17e2e659d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38973 } } peers { permanent_uuid: "0610f614ebd44277a007bea9a16e8d7b" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 43067 } attrs { promote: false } } peers { permanent_uuid: "93301179318146f7a4aec1331bd3df3f" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 46245 } attrs { promote: false } } }
I20260926 23:17:14.394170 24126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0610f614ebd44277a007bea9a16e8d7b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:17:14.410179 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.401811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:42235 (local address 127.0.0.1:52014)
0926 23:17:14.402275 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.402293 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:14.402417 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:14.402981 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:14.402986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:14.403015 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:14.403403 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.403417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.405078 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.405086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.407172 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.407189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.409138 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.409145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.409373 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.409548 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 23:17:14.409681 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 23:17:14.409831 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":114}
I20260926 23:17:14.410171 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.401988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.16.62.66:42235)
0926 23:17:14.402551 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:14.402558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:14.402580 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:17:14.402659 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:14.402667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:14.402752 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:14.402966 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:14.403637 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.404876 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.407426 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.408904 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.409066 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.409196 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:14.409716 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 23:17:14.409842 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":527,"threads_started":1}
I20260926 23:17:14.410923 23827 tablet_service.cc:1467] Processing CreateTablet for tablet 9161691003f9481e84d42a151e066e13 (DEFAULT_TABLE table=test-table [id=079589f292074ea1a0c416a626d20a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:14.411382 23827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9161691003f9481e84d42a151e066e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:14.415741 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.402298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35737 (local address 127.0.0.1:33820)
0926 23:17:14.405009 (+ 2711us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.405042 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:14.405160 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:14.406025 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:14.406032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:14.406066 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:14.406529 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.406543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.408816 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.408824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.412646 (+ 3822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.412663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.414989 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.414994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.415150 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.415173 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:17:14.415286 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 23:17:14.415403 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2561,"thread_start_us":12611,"threads_started":1}
I20260926 23:17:14.415746 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.402710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.16.62.65:35737)
0926 23:17:14.405667 (+ 2957us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:14.405674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:14.405694 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:17:14.405766 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:14.405772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:14.405864 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:14.405986 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:14.406900 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.408630 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.412890 (+ 4260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.414540 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.414708 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.414847 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:14.415311 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 23:17:14.415443 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2817,"thread_start_us":2679,"threads_started":1}
I20260926 23:17:14.416632 23682 tablet_service.cc:1467] Processing CreateTablet for tablet 9161691003f9481e84d42a151e066e13 (DEFAULT_TABLE table=test-table [id=079589f292074ea1a0c416a626d20a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:14.417049 23682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9161691003f9481e84d42a151e066e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:14.417629 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.402788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41175 (local address 127.0.0.1:57454)
0926 23:17:14.410394 (+ 7606us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.410412 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:14.410516 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:14.411709 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:14.411714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:14.411737 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:14.411912 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.411921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.413324 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.413331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.415189 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.415202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.416891 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.416897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.417124 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.417151 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 23:17:14.417235 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 23:17:14.417351 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7513,"thread_start_us":1927,"threads_started":1}
I20260926 23:17:14.417856 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.403094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.16.62.67:41175)
0926 23:17:14.404540 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:14.404547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:14.410933 (+ 6386us) server_negotiation.cc:407] Connection header received
0926 23:17:14.411341 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:14.411348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:14.411432 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:14.411527 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:14.412082 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.413173 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.415369 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.416604 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.416723 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.416812 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:14.417283 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 23:17:14.417409 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":1203,"threads_started":1}
I20260926 23:17:14.418674 24137 tablet_bootstrap.cc:492] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: Bootstrap starting.
I20260926 23:17:14.418675 23968 tablet_service.cc:1467] Processing CreateTablet for tablet 9161691003f9481e84d42a151e066e13 (DEFAULT_TABLE table=test-table [id=079589f292074ea1a0c416a626d20a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:14.419066 23968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9161691003f9481e84d42a151e066e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:14.420364 24137 tablet_bootstrap.cc:654] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:14.422436 24137 tablet_bootstrap.cc:492] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: No bootstrap required, opened a new log
I20260926 23:17:14.422565 24137 ts_tablet_manager.cc:1397] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 23:17:14.423457 24139 tablet_bootstrap.cc:492] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: Bootstrap starting.
I20260926 23:17:14.423375 24137 raft_consensus.cc:348] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } }
I20260926 23:17:14.423746 24137 raft_consensus.cc:374] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:14.423882 24137 raft_consensus.cc:729] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d13abca8344a939dc70777ed62c2f4, State: Initialized, Role: FOLLOWER
I20260926 23:17:14.424196 24137 consensus_queue.cc:260] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } }
I20260926 23:17:14.424649 24140 tablet_bootstrap.cc:492] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: Bootstrap starting.
I20260926 23:17:14.425350 24139 tablet_bootstrap.cc:654] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:14.426867 24140 tablet_bootstrap.cc:654] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:14.426965 24137 ts_tablet_manager.cc:1428] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260926 23:17:14.427846 24139 tablet_bootstrap.cc:492] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: No bootstrap required, opened a new log
I20260926 23:17:14.427985 24139 ts_tablet_manager.cc:1397] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 23:17:14.428738 24139 raft_consensus.cc:348] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } }
I20260926 23:17:14.428980 24139 raft_consensus.cc:374] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:14.429057 24139 raft_consensus.cc:729] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aba1825270a44f091e6ed417461c3a3, State: Initialized, Role: FOLLOWER
I20260926 23:17:14.429546 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.427494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.66:53153)
0926 23:17:14.429300 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.429391 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1697}
I20260926 23:17:14.429929 23892 heartbeater.cc:503] Master 127.16.62.124:46245 was elected leader, sending a full tablet report...
I20260926 23:17:14.429888 24139 consensus_queue.cc:260] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } }
I20260926 23:17:14.430612 24139 ts_tablet_manager.cc:1428] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 23:17:14.432449 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.431957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:54969)
0926 23:17:14.432208 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.432298 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 23:17:14.433727 23747 heartbeater.cc:503] Master 127.16.62.124:46245 was elected leader, sending a full tablet report...
I20260926 23:17:14.434504 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.431349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.66:32959)
0926 23:17:14.434252 (+ 2903us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.434349 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2794}
I20260926 23:17:14.435019 24140 tablet_bootstrap.cc:492] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: No bootstrap required, opened a new log
I20260926 23:17:14.435163 24140 ts_tablet_manager.cc:1397] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260926 23:17:14.435549 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.435168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:35933)
0926 23:17:14.435306 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.435390 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 23:17:14.436018 24140 raft_consensus.cc:348] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } }
I20260926 23:17:14.436267 24140 raft_consensus.cc:374] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:14.436364 24140 raft_consensus.cc:729] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccd0d1f512249ebaed630f09bbc3c6f, State: Initialized, Role: FOLLOWER
I20260926 23:17:14.436630 24140 consensus_queue.cc:260] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } }
I20260926 23:17:14.437304 24140 ts_tablet_manager.cc:1428] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:17:14.437943 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.437570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.67:35409)
0926 23:17:14.437717 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.437800 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 23:17:14.438938 24033 heartbeater.cc:503] Master 127.16.62.124:46245 was elected leader, sending a full tablet report...
I20260926 23:17:14.439484 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.439052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.67:60433)
0926 23:17:14.439271 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.439343 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260926 23:17:14.520262 23895 tablet.cc:2369] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:17:14.568846 24144 raft_consensus.cc:482] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:14.569087 24144 raft_consensus.cc:504] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } }
I20260926 23:17:14.570552 24144 leader_election.cc:288] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0d13abca8344a939dc70777ed62c2f4 (127.16.62.66:42235), dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175)
I20260926 23:17:14.576628 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.570873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:44425 (local address 127.16.62.66:42235)
0926 23:17:14.571062 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:14.571066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:14.571184 (+ 118us) server_negotiation.cc:407] Connection header received
0926 23:17:14.571387 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:14.571394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:14.571477 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:14.571576 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:14.572318 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.573141 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.574875 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.575696 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.575884 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.575960 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:14.576254 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 23:17:14.576374 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:17:14.576670 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.570870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:42235 (local address 127.16.62.65:44425)
0926 23:17:14.571063 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.571082 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:14.571231 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:14.571614 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:14.571621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:14.571655 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:14.572162 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.572175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.573269 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.573277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.574743 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.574753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.575882 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.575887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.576060 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.576082 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:17:14.576221 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 23:17:14.576357 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":28}
I20260926 23:17:14.577392 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.571553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:56979 (local address 127.16.62.67:41175)
0926 23:17:14.572002 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:14.572007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:14.572300 (+ 293us) server_negotiation.cc:407] Connection header received
0926 23:17:14.572481 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:14.572486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:14.572551 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:14.572648 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:14.573127 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.573812 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.575644 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.576505 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.576680 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.576751 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:14.577031 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 23:17:14.577152 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260926 23:17:14.577495 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.571309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41175 (local address 127.16.62.65:56979)
0926 23:17:14.572211 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.572226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:14.572354 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:14.572667 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:14.572671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:14.572695 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:14.572993 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.573001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.573937 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.573947 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.575494 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:14.575505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.576718 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.576725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:14.576880 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.576902 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:17:14.577006 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 23:17:14.577120 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":20,"thread_start_us":442,"threads_started":1}
I20260926 23:17:14.577713 23847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9161691003f9481e84d42a151e066e13" candidate_uuid: "5aba1825270a44f091e6ed417461c3a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d13abca8344a939dc70777ed62c2f4" is_pre_election: true
I20260926 23:17:14.578161 23847 raft_consensus.cc:2454] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aba1825270a44f091e6ed417461c3a3 in term 0.
I20260926 23:17:14.578202 23988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9161691003f9481e84d42a151e066e13" candidate_uuid: "5aba1825270a44f091e6ed417461c3a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" is_pre_election: true
I20260926 23:17:14.578418 23988 raft_consensus.cc:2454] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aba1825270a44f091e6ed417461c3a3 in term 0.
I20260926 23:17:14.578703 23635 leader_election.cc:302] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aba1825270a44f091e6ed417461c3a3, b0d13abca8344a939dc70777ed62c2f4; no voters:
I20260926 23:17:14.579025 24144 raft_consensus.cc:2781] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:17:14.579154 24144 raft_consensus.cc:482] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:17:14.579257 24144 raft_consensus.cc:3037] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:14.581316 24144 raft_consensus.cc:504] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } }
I20260926 23:17:14.581845 24144 leader_election.cc:288] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [CANDIDATE]: Term 1 election: Requested vote from peers b0d13abca8344a939dc70777ed62c2f4 (127.16.62.66:42235), dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175)
I20260926 23:17:14.582224 23847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9161691003f9481e84d42a151e066e13" candidate_uuid: "5aba1825270a44f091e6ed417461c3a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d13abca8344a939dc70777ed62c2f4"
I20260926 23:17:14.582312 23988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9161691003f9481e84d42a151e066e13" candidate_uuid: "5aba1825270a44f091e6ed417461c3a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccd0d1f512249ebaed630f09bbc3c6f"
I20260926 23:17:14.582420 23847 raft_consensus.cc:3037] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:14.582472 23988 raft_consensus.cc:3037] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:14.585413 23988 raft_consensus.cc:2454] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aba1825270a44f091e6ed417461c3a3 in term 1.
I20260926 23:17:14.585413 23847 raft_consensus.cc:2454] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aba1825270a44f091e6ed417461c3a3 in term 1.
I20260926 23:17:14.585819 23635 leader_election.cc:302] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aba1825270a44f091e6ed417461c3a3, b0d13abca8344a939dc70777ed62c2f4; no voters:
I20260926 23:17:14.586069 24144 raft_consensus.cc:2781] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:14.586503 24144 raft_consensus.cc:686] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 1 LEADER]: Becoming Leader. State: Replica: 5aba1825270a44f091e6ed417461c3a3, State: Running, Role: LEADER
I20260926 23:17:14.586941 24144 consensus_queue.cc:237] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } }
I20260926 23:17:14.587476 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.587105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:40857)
0926 23:17:14.587269 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.587374 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 23:17:14.590646 23569 catalog_manager.cc:5495] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aba1825270a44f091e6ed417461c3a3 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "5aba1825270a44f091e6ed417461c3a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aba1825270a44f091e6ed417461c3a3" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 } health_report { overall_health: UNKNOWN } } }
I20260926 23:17:14.647672 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.642956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.16.62.65:35737)
0926 23:17:14.643141 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:14.643145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:14.643162 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:17:14.643282 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:14.643286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:14.643355 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:14.643484 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:14.643793 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.644529 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.645709 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.646110 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.646200 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.646706 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:14.647198 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:14.647312 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:14.647412 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 23:17:14.647478 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:17:14.651494 23988 raft_consensus.cc:1261] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [term 1 FOLLOWER]: Refusing update from remote peer 5aba1825270a44f091e6ed417461c3a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:14.651810 23847 raft_consensus.cc:1261] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [term 1 FOLLOWER]: Refusing update from remote peer 5aba1825270a44f091e6ed417461c3a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:14.652267 24147 consensus_queue.cc:1035] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dccd0d1f512249ebaed630f09bbc3c6f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:14.652637 24144 consensus_queue.cc:1035] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d13abca8344a939dc70777ed62c2f4" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:14.654986 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.654486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.66:40947)
0926 23:17:14.654682 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.654788 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 23:17:14.656020 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.655638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.67:36605)
0926 23:17:14.655818 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.655889 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 23:17:14.662186 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.661558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.66:45615)
0926 23:17:14.661816 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.661934 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 23:17:14.669629 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.668948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:55583)
0926 23:17:14.669252 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.669381 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 23:17:14.677908 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.677483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:60069)
0926 23:17:14.677621 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.677726 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 23:17:14.679359 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.678485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.65:32927)
0926 23:17:14.679035 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.679140 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
I20260926 23:17:14.681499 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.680906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.16.62.67:33423)
0926 23:17:14.681198 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:14.681309 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260926 23:17:14.693698 23569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:14.693941 23569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:14.699697 23569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93301179318146f7a4aec1331bd3df3f: Sending DeleteTablet for 3 replicas of tablet 9161691003f9481e84d42a151e066e13
I20260926 23:17:14.700744 23968 tablet_service.cc:1514] Processing DeleteTablet for tablet 9161691003f9481e84d42a151e066e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:14 UTC) from {username='oryx', principal='oryx/127.16.62.124@KRBTEST.COM'} at 127.0.0.1:57454
I20260926 23:17:14.700975 23827 tablet_service.cc:1514] Processing DeleteTablet for tablet 9161691003f9481e84d42a151e066e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:14 UTC) from {username='oryx', principal='oryx/127.16.62.124@KRBTEST.COM'} at 127.0.0.1:52014
I20260926 23:17:14.701205 23682 tablet_service.cc:1514] Processing DeleteTablet for tablet 9161691003f9481e84d42a151e066e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:14 UTC) from {username='oryx', principal='oryx/127.16.62.124@KRBTEST.COM'} at 127.0.0.1:33820
I20260926 23:17:14.701561 24159 tablet_replica.cc:330] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: stopping tablet replica
I20260926 23:17:14.701790 24159 raft_consensus.cc:2229] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:14.701905 24161 tablet_replica.cc:330] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: stopping tablet replica
I20260926 23:17:14.702116 24159 raft_consensus.cc:2258] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:14.702085 24161 raft_consensus.cc:2229] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:17:14.702687 24161 raft_consensus.cc:2258] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:14.703099 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 23541
I20260926 23:17:14.705240 24161 ts_tablet_manager.cc:1905] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:14.711750 24160 tablet_replica.cc:330] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: stopping tablet replica
I20260926 23:17:14.711946 24160 raft_consensus.cc:2229] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:14.712280 24160 raft_consensus.cc:2258] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:14.712604 24159 ts_tablet_manager.cc:1905] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:14.713425 24160 ts_tablet_manager.cc:1905] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:14.714135 24161 ts_tablet_manager.cc:1918] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:14.714319 24161 log.cc:1196] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/9161691003f9481e84d42a151e066e13
I20260926 23:17:14.714874 24161 ts_tablet_manager.cc:1939] T 9161691003f9481e84d42a151e066e13 P 5aba1825270a44f091e6ed417461c3a3: Deleting consensus metadata
I20260926 23:17:14.716069 23556 catalog_manager.cc:4841] TS 5aba1825270a44f091e6ed417461c3a3 (127.16.62.65:35737): tablet 9161691003f9481e84d42a151e066e13 (table test-table [id=079589f292074ea1a0c416a626d20a36]) successfully deleted
I20260926 23:17:14.718863 24160 ts_tablet_manager.cc:1918] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:14.719002 24160 log.cc:1196] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/9161691003f9481e84d42a151e066e13
I20260926 23:17:14.719388 24160 ts_tablet_manager.cc:1939] T 9161691003f9481e84d42a151e066e13 P b0d13abca8344a939dc70777ed62c2f4: Deleting consensus metadata
I20260926 23:17:14.720675 23553 catalog_manager.cc:4841] TS b0d13abca8344a939dc70777ed62c2f4 (127.16.62.66:42235): tablet 9161691003f9481e84d42a151e066e13 (table test-table [id=079589f292074ea1a0c416a626d20a36]) successfully deleted
I20260926 23:17:14.723323 24159 ts_tablet_manager.cc:1918] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:14.723467 24159 log.cc:1196] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/9161691003f9481e84d42a151e066e13
I20260926 23:17:14.723883 24159 ts_tablet_manager.cc:1939] T 9161691003f9481e84d42a151e066e13 P dccd0d1f512249ebaed630f09bbc3c6f: Deleting consensus metadata
I20260926 23:17:14.724967 23553 catalog_manager.cc:4841] TS dccd0d1f512249ebaed630f09bbc3c6f (127.16.62.67:41175): tablet 9161691003f9481e84d42a151e066e13 (table test-table [id=079589f292074ea1a0c416a626d20a36]) successfully deleted
I20260926 23:17:14.733350 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.705838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.16.62.65:35737)
0926 23:17:14.706132 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:14.706138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:14.718300 (+ 12162us) server_negotiation.cc:407] Connection header received
0926 23:17:14.718746 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:14.718753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:14.718869 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:14.718944 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:14.721040 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.722217 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.724106 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.724756 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.724910 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.727487 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:14.727499 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:14.727525 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:14.727560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:14.730479 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:14.731098 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:14.731107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:14.731110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:14.731265 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:14.731650 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:14.731655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:14.731659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:14.732059 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:14.732419 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:14.732734 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 23:17:14.732968 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260926 23:17:14.734225 23742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:14.840991 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 23541
I20260926 23:17:14.859884 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 23759
I20260926 23:17:14.872666 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.860603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.16.62.66:42235)
0926 23:17:14.860821 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:14.860825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:14.860895 (+ 70us) server_negotiation.cc:407] Connection header received
0926 23:17:14.861043 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:14.861047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:14.861113 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:14.861195 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:14.862448 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.863238 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.864565 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:14.864947 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:14.865036 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:14.867301 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:14.867315 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:14.867330 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:14.867358 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:14.870041 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:14.870619 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:14.870625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:14.870627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:14.870745 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:14.871043 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:14.871047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:14.871048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:14.871362 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:14.871569 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:14.871990 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 23:17:14.872419 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 23:17:14.873344 23887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:14.990695 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 23759
I20260926 23:17:15.000862 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:14.999982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44684)
0926 23:17:15.000603 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:15.000709 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":410,"threads_started":1}
I20260926 23:17:15.002115 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:15.001800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:38973 (local address 127.0.0.1:44696)
0926 23:17:15.001929 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:15.002002 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 23:17:15.012728 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 23905
I20260926 23:17:15.026117 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:15.013454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.16.62.67:41175)
0926 23:17:15.013625 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:15.013633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:15.013689 (+ 56us) server_negotiation.cc:407] Connection header received
0926 23:17:15.013897 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:15.013907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:15.013977 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:15.014080 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:15.015397 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:15.016128 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:15.017514 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:15.017984 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:15.018117 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:15.020818 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:15.020835 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:15.020862 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:15.020912 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:15.023407 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:15.024051 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:15.024060 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:15.024064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:15.024267 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:15.024620 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:15.024625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:15.024628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:15.024989 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:15.025319 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:15.025581 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 23:17:15.025795 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:17:15.026748 24028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:15.119921 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 23905
I20260926 23:17:15.136929 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 23243
I20260926 23:17:15.137995 23409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:15.254289 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 23243
I20260926 23:17:15.274799 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 23343
I20260926 23:17:15.275875 23599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:15.393285 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 23343
2026-09-26T23:17:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10982 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:17:15 dist-test-slave-vcv1 krb5kdc[24172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:17:15 dist-test-slave-vcv1 krb5kdc[24172](info): set up 2 sockets
Sep 26 23:17:15 dist-test-slave-vcv1 krb5kdc[24172](info): commencing operation
krb5kdc: starting...
W20260926 23:17:16.481767 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:17:16 dist-test-slave-vcv1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:17:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.126.keytab.
Entry for principal oryx/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.126.keytab.
I20260926 23:17:16.683537 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:37855
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.126.keytab
--principal=oryx/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:37855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:16.999174 24185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:16.999440 24185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:16.999526 24185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:17.006626 24185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:17.006745 24185 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 23:17:17.006784 24185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:17.006834 24185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:17.006873 24185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:17.006932 24185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:17.016353 24185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:37855
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.126.keytab
--principal=oryx/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:17.017503 24185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:17.018692 24185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:17.029946 24190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:17.029994 24193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:17.030032 24191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:17.030896 24185 server_base.cc:1034] running on GCE node
I20260926 23:17:17.031531 24185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:17.032831 24185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:17.034041 24185 hybrid_clock.cc:648] HybridClock initialized: now 1790464637033975 us; error 38 us; skew 500 ppm
I20260926 23:17:17.034322 24185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:17 dist-test-slave-vcv1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464637, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:17.037297 24185 init.cc:377] Logged in from keytab as oryx/127.16.62.126@KRBTEST.COM (short username oryx)
I20260926 23:17:17.038460 24185 webserver.cc:460] Webserver started at http://127.16.62.126:44573/ using document root <none> and password file <none>
I20260926 23:17:17.039031 24185 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:17.039106 24185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:17.039372 24185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:17.042080 24185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "8eb64b9e04f04017a4c3fa465233bc4d"
format_stamp: "Formatted at 2026-09-26 23:17:17 on dist-test-slave-vcv1"
server_key: "6f2534c692a5c171c582565f79cc213f"
server_key_iv: "dd703e1ba39265d7b5d6e32a4ce33990"
server_key_version: "encryptionkey@0"
I20260926 23:17:17.042769 24185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "8eb64b9e04f04017a4c3fa465233bc4d"
format_stamp: "Formatted at 2026-09-26 23:17:17 on dist-test-slave-vcv1"
server_key: "6f2534c692a5c171c582565f79cc213f"
server_key_iv: "dd703e1ba39265d7b5d6e32a4ce33990"
server_key_version: "encryptionkey@0"
I20260926 23:17:17.047456 24185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:17:17.050621 24200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:17.051863 24185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:17:17.051985 24185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "8eb64b9e04f04017a4c3fa465233bc4d"
format_stamp: "Formatted at 2026-09-26 23:17:17 on dist-test-slave-vcv1"
server_key: "6f2534c692a5c171c582565f79cc213f"
server_key_iv: "dd703e1ba39265d7b5d6e32a4ce33990"
server_key_version: "encryptionkey@0"
I20260926 23:17:17.052197 24185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:17.070459 24185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:17.074129 24185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:17.074347 24185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:17.095211 24185 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:37855
I20260926 23:17:17.095248 24261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:37855 every 8 connection(s)
I20260926 23:17:17.096540 24185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:17:17.100121 24262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:17.101595 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 24185
I20260926 23:17:17.101819 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:17:17.102603 16633 external_mini_cluster.cc:1428] Setting key 450f1eecb88feb5befa87c7553e60b15
I20260926 23:17:17.113914 24262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d: Bootstrap starting.
Sep 26 23:17:17 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:17.117486 24262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:17.118605 24262 log.cc:824] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:17.121260 24262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d: No bootstrap required, opened a new log
I20260926 23:17:17.124490 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:17.104597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.16.62.126:37855)
0926 23:17:17.105336 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:17.105359 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:17.105394 (+ 35us) server_negotiation.cc:407] Connection header received
0926 23:17:17.106793 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:17.106811 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:17.107162 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:17.107449 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:17.109132 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.110366 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.112222 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.112838 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.112951 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:17.117417 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:17.117426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:17.117439 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:17.117470 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:17.120699 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:17.121329 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:17.121335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:17.121340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:17.121545 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:17.121934 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:17.121938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:17.121942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:17.122398 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:17.122701 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:17.123131 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 23:17:17.123510 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":339,"threads_started":1}
I20260926 23:17:17.128929 24262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb64b9e04f04017a4c3fa465233bc4d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37855 } }
I20260926 23:17:17.129254 24262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:17.129354 24262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb64b9e04f04017a4c3fa465233bc4d, State: Initialized, Role: FOLLOWER
I20260926 23:17:17.129846 24262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb64b9e04f04017a4c3fa465233bc4d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37855 } }
I20260926 23:17:17.130080 24262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:17:17.130231 24262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:17:17.130391 24262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:17.132880 24262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb64b9e04f04017a4c3fa465233bc4d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37855 } }
I20260926 23:17:17.133380 24262 leader_election.cc:302] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eb64b9e04f04017a4c3fa465233bc4d; no voters:
I20260926 23:17:17.133904 24262 leader_election.cc:288] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:17:17.134146 24267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:17.134896 24267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [term 1 LEADER]: Becoming Leader. State: Replica: 8eb64b9e04f04017a4c3fa465233bc4d, State: Running, Role: LEADER
I20260926 23:17:17.135490 24267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb64b9e04f04017a4c3fa465233bc4d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37855 } }
I20260926 23:17:17.135984 24262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:17.138319 24268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eb64b9e04f04017a4c3fa465233bc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb64b9e04f04017a4c3fa465233bc4d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37855 } } }
I20260926 23:17:17.138700 24268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:17.138623 24269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb64b9e04f04017a4c3fa465233bc4d. Latest consensus state: current_term: 1 leader_uuid: "8eb64b9e04f04017a4c3fa465233bc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb64b9e04f04017a4c3fa465233bc4d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37855 } } }
I20260926 23:17:17.138962 24269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:17.141162 24274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:17:17.146380 24274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:17:17.154419 24274 catalog_manager.cc:1347] Generated new cluster ID: 390185b01eae46039ab1e987a871694f
I20260926 23:17:17.154541 24274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:17:17.165444 24274 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:17:17.167181 24274 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:17:17.174942 24274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d: Generated new TSK 0
I20260926 23:17:17.175890 24274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab.
Entry for principal oryx/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab.
I20260926 23:17:17.228209 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:37855
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab
--principal=oryx/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:17.544070 24288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:17.544401 24288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:17.544493 24288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:17.551566 24288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:17.551661 24288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:17.551739 24288 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:17:17.551815 24288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:17:17.561501 24288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab
--principal=oryx/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.62.126:37855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:17.562654 24288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:17.563902 24288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:17.577147 24294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:17.577176 24293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:17.578047 24296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:17.578119 24288 server_base.cc:1034] running on GCE node
I20260926 23:17:17.578788 24288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:17.579713 24288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:17.580978 24288 hybrid_clock.cc:648] HybridClock initialized: now 1790464637580912 us; error 38 us; skew 500 ppm
I20260926 23:17:17.581324 24288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:17 dist-test-slave-vcv1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464637, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:17.584612 24288 init.cc:377] Logged in from keytab as oryx/127.16.62.65@KRBTEST.COM (short username oryx)
I20260926 23:17:17.586097 24288 webserver.cc:460] Webserver started at http://127.16.62.65:33121/ using document root <none> and password file <none>
I20260926 23:17:17.586753 24288 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:17.586838 24288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:17.587106 24288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:17.589948 24288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "3ab65987a9084ac2a96c0a977c774418"
format_stamp: "Formatted at 2026-09-26 23:17:17 on dist-test-slave-vcv1"
server_key: "c09c44d463a1ed1749ebff9b7cbfa663"
server_key_iv: "47176f1947370f05b9bd1803440c3d9b"
server_key_version: "encryptionkey@0"
I20260926 23:17:17.590664 24288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "3ab65987a9084ac2a96c0a977c774418"
format_stamp: "Formatted at 2026-09-26 23:17:17 on dist-test-slave-vcv1"
server_key: "c09c44d463a1ed1749ebff9b7cbfa663"
server_key_iv: "47176f1947370f05b9bd1803440c3d9b"
server_key_version: "encryptionkey@0"
I20260926 23:17:17.595571 24288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:17:17.598980 24303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:17.600505 24288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:17.600648 24288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "3ab65987a9084ac2a96c0a977c774418"
format_stamp: "Formatted at 2026-09-26 23:17:17 on dist-test-slave-vcv1"
server_key: "c09c44d463a1ed1749ebff9b7cbfa663"
server_key_iv: "47176f1947370f05b9bd1803440c3d9b"
server_key_version: "encryptionkey@0"
I20260926 23:17:17.600811 24288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:17.617003 24288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:17.620997 24288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:17.621228 24288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:17.623625 24288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:17.623775 24288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:17:17.623948 24288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:17.624068 24288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:17:17 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464637, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:17.648046 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:17.623946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:42307 (local address 127.16.62.126:37855)
0926 23:17:17.624191 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:17.624197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:17.625471 (+ 1274us) server_negotiation.cc:407] Connection header received
0926 23:17:17.628392 (+ 2921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:17.628397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:17.628496 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:17.628632 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:17.631096 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.631991 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.635271 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.635787 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.635910 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:17.641649 (+ 5739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:17.641654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:17.641657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:17.641688 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:17.644178 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:17.645143 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:17.645151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:17.645156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:17.645280 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:17.645846 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:17.645854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:17.645860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:17.646311 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:17.646498 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:17.647267 (+ 769us) server_negotiation.cc:299] Negotiation successful
0926 23:17:17.647551 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:17:17.648665 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:17.624442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37855 (local address 127.16.62.65:42307)
0926 23:17:17.625282 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:17.625338 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:17.628058 (+ 2720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:17.628882 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:17.628900 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:17.629778 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:17.630823 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:17.630851 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.632186 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.632195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:17.635100 (+ 2905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:17.635120 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.636062 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.636072 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:17.636484 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:17.637719 (+ 1235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:17.637750 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:17.641424 (+ 3674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:17.644345 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:17.644353 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:17.644367 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:17.645012 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:17.645392 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:17.645397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:17.645401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:17.645674 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:17.646446 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:17.646451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:17.646759 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 23:17:17.647122 (+ 363us) client_negotiation.cc:240] Negotiation successful
0926 23:17:17.647502 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":315,"threads_started":1}
I20260926 23:17:17.671157 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:17.654366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37855 (local address 127.16.62.65:44165)
0926 23:17:17.655158 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:17.655189 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:17.655425 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:17.655777 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:17.655785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:17.656483 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:17.657279 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:17.657296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.658566 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.658574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:17.660910 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:17.660928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.662118 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.662129 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:17.662382 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:17.663075 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:17.663104 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:17.664522 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:17.668041 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:17.668046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:17.668052 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:17.668855 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:17.669276 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:17.669283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:17.669287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:17.669494 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:17.670340 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:17.670345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:17.670503 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 23:17:17.670610 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 23:17:17.670832 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":405,"threads_started":1}
I20260926 23:17:17.671567 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:17.654392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:44165 (local address 127.16.62.126:37855)
0926 23:17:17.654576 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:17.654583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:17.655351 (+ 768us) server_negotiation.cc:407] Connection header received
0926 23:17:17.655557 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:17.655562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:17.655626 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:17.655705 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:17.657458 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.658385 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.661089 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:17.661659 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:17.661785 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:17.664716 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:17.664721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:17.664724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:17.664758 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:17.667872 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:17.669002 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:17.669009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:17.669012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:17.669160 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:17.669643 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:17.669651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:17.669656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:17.670203 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:17.670370 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:17.670790 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 23:17:17.671069 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 23:17:17.678308 24288 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:42319
I20260926 23:17:17.678350 24422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:42319 every 8 connection(s)
I20260926 23:17:17.679464 24288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:17:17.681255 24423 heartbeater.cc:347] Connected to a master server at 127.16.62.126:37855
I20260926 23:17:17.681474 24423 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:17.682076 24423 heartbeater.cc:511] Master 127.16.62.126:37855 requested a full tablet report, sending...
I20260926 23:17:17.683928 24217 ts_manager.cc:194] Registered new tserver with Master: 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319)
I20260926 23:17:17.685321 24217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:42307
I20260926 23:17:17.687742 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 24288
I20260926 23:17:17.687959 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:17:17.688629 16633 external_mini_cluster.cc:1428] Setting key eab66efe498bc73d63c1d5b156958c49
WARNING: no policy specified for oryx/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab.
Entry for principal oryx/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab.
I20260926 23:17:17.732554 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:37855
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab
--principal=oryx/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:18.052424 24427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:18.052690 24427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:18.052917 24427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:18.060036 24427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:18.060156 24427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:18.060320 24427 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:17:18.060434 24427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:17:18.069679 24427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab
--principal=oryx/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.62.126:37855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:18.070930 24427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:18.072330 24427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:18.085093 24433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:18.085139 24432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:18.085037 24435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:18.085930 24427 server_base.cc:1034] running on GCE node
I20260926 23:17:18.086583 24427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:18.087414 24427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:18.088634 24427 hybrid_clock.cc:648] HybridClock initialized: now 1790464638088569 us; error 54 us; skew 500 ppm
I20260926 23:17:18.088898 24427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:18 dist-test-slave-vcv1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464638, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:18.091885 24427 init.cc:377] Logged in from keytab as oryx/127.16.62.66@KRBTEST.COM (short username oryx)
I20260926 23:17:18.093422 24427 webserver.cc:460] Webserver started at http://127.16.62.66:37703/ using document root <none> and password file <none>
I20260926 23:17:18.094074 24427 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:18.094210 24427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:18.094581 24427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:18.098310 24427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032"
format_stamp: "Formatted at 2026-09-26 23:17:18 on dist-test-slave-vcv1"
server_key: "83d03284f5363fbb858de3fa782199ab"
server_key_iv: "0861a3a210fbfbf84827b00469036efa"
server_key_version: "encryptionkey@0"
I20260926 23:17:18.099241 24427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032"
format_stamp: "Formatted at 2026-09-26 23:17:18 on dist-test-slave-vcv1"
server_key: "83d03284f5363fbb858de3fa782199ab"
server_key_iv: "0861a3a210fbfbf84827b00469036efa"
server_key_version: "encryptionkey@0"
I20260926 23:17:18.105144 24427 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 23:17:18.108959 24442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:18.110502 24427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:17:18.110644 24427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032"
format_stamp: "Formatted at 2026-09-26 23:17:18 on dist-test-slave-vcv1"
server_key: "83d03284f5363fbb858de3fa782199ab"
server_key_iv: "0861a3a210fbfbf84827b00469036efa"
server_key_version: "encryptionkey@0"
I20260926 23:17:18.110849 24427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:18.148469 24427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:18.153030 24427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:18.153263 24427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:18.155864 24427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:18.155967 24427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:18.156162 24427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:18.156235 24427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 26 23:17:18 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464638, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:18.180697 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:18.156528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:37125 (local address 127.16.62.126:37855)
0926 23:17:18.156706 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:18.156712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:18.158151 (+ 1439us) server_negotiation.cc:407] Connection header received
0926 23:17:18.161127 (+ 2976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:18.161133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:18.161219 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:18.161310 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:18.164281 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.165293 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.167661 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.168397 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.168539 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:18.173588 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:18.173595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:18.173600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:18.173643 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:18.176969 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:18.177968 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:18.177975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:18.177979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:18.178158 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:18.178666 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:18.178675 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:18.178681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:18.179147 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:18.179353 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:18.179958 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 23:17:18.180264 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:17:18.181623 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:18.157073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37855 (local address 127.16.62.66:37125)
0926 23:17:18.157948 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:18.158014 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:18.160766 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:18.161946 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:18.161961 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:18.162989 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:18.164044 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:18.164077 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.165439 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.165445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:18.167498 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:18.167509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.168692 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.168703 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:18.169083 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:18.170046 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:18.170105 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:18.173367 (+ 3262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:18.177167 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:18.177178 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:18.177196 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:18.177851 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:18.178302 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:18.178313 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:18.178316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:18.178518 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:18.179296 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:18.179301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:18.179598 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 23:17:18.179810 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 23:17:18.180176 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":386,"threads_started":1}
I20260926 23:17:18.212563 24427 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:33863
I20260926 23:17:18.212652 24561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:33863 every 8 connection(s)
I20260926 23:17:18.213119 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:18.196016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:46133 (local address 127.16.62.126:37855)
0926 23:17:18.196288 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:18.196297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:18.197166 (+ 869us) server_negotiation.cc:407] Connection header received
0926 23:17:18.197376 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:18.197382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:18.197478 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:18.197591 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:18.198652 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.199963 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.201382 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.202168 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.202310 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:18.205328 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:18.205337 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:18.205343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:18.205389 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:18.209597 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:18.210608 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:18.210613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:18.210615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:18.210751 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:18.211212 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:18.211220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:18.211226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:18.211703 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:18.211873 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:18.212494 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 23:17:18.212756 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:17:18.213213 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:18.196005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37855 (local address 127.16.62.66:46133)
0926 23:17:18.197029 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:18.197056 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:18.197234 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:18.197651 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:18.197658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:18.198168 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:18.198481 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:18.198490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.200111 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.200116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:18.201244 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:18.201254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.202620 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.202633 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:18.202899 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:18.203624 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:18.203654 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:18.204992 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:18.209780 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:18.209791 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:18.209798 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:18.210462 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:18.210864 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:18.210872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:18.210876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:18.211034 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:18.211853 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:18.211862 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:18.212112 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 23:17:18.212317 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 23:17:18.212749 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":705,"threads_started":1}
I20260926 23:17:18.214078 24427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:17:18.215664 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 24427
I20260926 23:17:18.215868 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:17:18.216464 16633 external_mini_cluster.cc:1428] Setting key a9fa18aedf1c1591afa7c9d0520bb381
I20260926 23:17:18.216333 24562 heartbeater.cc:347] Connected to a master server at 127.16.62.126:37855
I20260926 23:17:18.217207 24562 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:18.217976 24562 heartbeater.cc:511] Master 127.16.62.126:37855 requested a full tablet report, sending...
I20260926 23:17:18.219295 24217 ts_manager.cc:194] Registered new tserver with Master: a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863)
I20260926 23:17:18.220101 24217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:37125
WARNING: no policy specified for oryx/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab.
Entry for principal oryx/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab.
I20260926 23:17:18.261027 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:37855
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab
--principal=oryx/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:18.570699 24566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:18.570976 24566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:18.571069 24566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:18.578299 24566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:18.578403 24566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:18.578485 24566 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:17:18.578565 24566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:17:18.588023 24566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab
--principal=oryx/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.62.126:37855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:18.589221 24566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:18.590483 24566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:18.602588 24571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:18.602804 24572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:18.602952 24574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:18.603891 24566 server_base.cc:1034] running on GCE node
I20260926 23:17:18.604600 24566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:18.605470 24566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:18.606724 24566 hybrid_clock.cc:648] HybridClock initialized: now 1790464638606651 us; error 49 us; skew 500 ppm
I20260926 23:17:18.607016 24566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:18 dist-test-slave-vcv1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464638, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:18.609982 24566 init.cc:377] Logged in from keytab as oryx/127.16.62.67@KRBTEST.COM (short username oryx)
I20260926 23:17:18.611240 24566 webserver.cc:460] Webserver started at http://127.16.62.67:41987/ using document root <none> and password file <none>
I20260926 23:17:18.611883 24566 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:18.611991 24566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:18.612358 24566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:18.615134 24566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "896629842d4048a7b51cbcd4af8f7252"
format_stamp: "Formatted at 2026-09-26 23:17:18 on dist-test-slave-vcv1"
server_key: "5574e53efaa2f0487a526b168b20e148"
server_key_iv: "73fd9d6c53ea81f1f13fc3596397e415"
server_key_version: "encryptionkey@0"
I20260926 23:17:18.616009 24566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "896629842d4048a7b51cbcd4af8f7252"
format_stamp: "Formatted at 2026-09-26 23:17:18 on dist-test-slave-vcv1"
server_key: "5574e53efaa2f0487a526b168b20e148"
server_key_iv: "73fd9d6c53ea81f1f13fc3596397e415"
server_key_version: "encryptionkey@0"
I20260926 23:17:18.620723 24566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:17:18.623905 24581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:18.625273 24566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:18.625434 24566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "896629842d4048a7b51cbcd4af8f7252"
format_stamp: "Formatted at 2026-09-26 23:17:18 on dist-test-slave-vcv1"
server_key: "5574e53efaa2f0487a526b168b20e148"
server_key_iv: "73fd9d6c53ea81f1f13fc3596397e415"
server_key_version: "encryptionkey@0"
I20260926 23:17:18.625658 24566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:18.642407 24566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:18.646183 24566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:18.646443 24566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:18.649114 24566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:18.649232 24566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:18.649406 24566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:18.649487 24566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:17:18 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464638, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:18.672281 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:18.649364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:39737 (local address 127.16.62.126:37855)
0926 23:17:18.649545 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:18.649551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:18.651163 (+ 1612us) server_negotiation.cc:407] Connection header received
0926 23:17:18.653211 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:18.653216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:18.653313 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:18.653424 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:18.656015 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.656898 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.660276 (+ 3378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.660870 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.661010 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:18.665908 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:18.665913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:18.665916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:18.665947 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:18.668566 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:18.669538 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:18.669543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:18.669547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:18.669728 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:18.670277 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:18.670282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:18.670286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:18.670656 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:18.670805 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:18.671690 (+ 885us) server_negotiation.cc:299] Negotiation successful
0926 23:17:18.671901 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:17:18.673537 24590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:18.650032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37855 (local address 127.16.62.67:39737)
0926 23:17:18.650973 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:18.651018 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:18.652941 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:18.653733 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:18.653751 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:18.654649 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:18.655746 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:18.655801 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.657070 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.657082 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:18.660068 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:18.660088 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.661177 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.661187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:18.661579 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:18.662468 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:18.662508 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:18.665685 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:18.668703 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:18.668715 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:18.668729 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:18.669386 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:18.669918 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:18.669942 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:18.669945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:18.670151 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:18.670789 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:18.670796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:18.671311 (+ 515us) client_negotiation.cc:769] Sending connection context
0926 23:17:18.671606 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 23:17:18.672093 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":417,"threads_started":1}
I20260926 23:17:18.687769 24423 heartbeater.cc:503] Master 127.16.62.126:37855 was elected leader, sending a full tablet report...
I20260926 23:17:18.703862 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:18.680596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37855 (local address 127.16.62.67:46783)
0926 23:17:18.688791 (+ 8195us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:18.688815 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:18.688973 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:18.689472 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:18.689476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:18.690205 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:18.690672 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:18.690689 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.692039 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.692046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:18.693821 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:18.693842 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.695001 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.695012 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:18.695221 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:18.695905 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:18.695934 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:18.697213 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:18.700841 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:18.700846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:18.700850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:18.701224 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:18.701694 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:18.701699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:18.701702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:18.701833 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:18.702454 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:18.702458 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:18.702663 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 23:17:18.702952 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 23:17:18.703290 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7999,"thread_start_us":7840,"threads_started":1}
I20260926 23:17:18.704883 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:18.680607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:46783 (local address 127.16.62.126:37855)
0926 23:17:18.680813 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:18.680820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:18.688924 (+ 8104us) server_negotiation.cc:407] Connection header received
0926 23:17:18.689104 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:18.689108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:18.689205 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:18.689301 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:18.690862 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.691837 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.694022 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.694662 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.694733 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:18.697437 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:18.697443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:18.697448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:18.697487 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:18.700713 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:18.701349 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:18.701356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:18.701362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:18.701556 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:18.702026 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:18.702032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:18.702037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:18.702330 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:18.702482 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:18.704219 (+ 1737us) server_negotiation.cc:299] Negotiation successful
0926 23:17:18.704430 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:17:18.707252 24566 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:46007
I20260926 23:17:18.707316 24700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:46007 every 8 connection(s)
I20260926 23:17:18.708829 24566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:17:18.709625 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 24566
I20260926 23:17:18.709853 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:17:18.710389 16633 external_mini_cluster.cc:1428] Setting key 7f5ecf14d088da625078413ca10acb62
I20260926 23:17:18.711066 24701 heartbeater.cc:347] Connected to a master server at 127.16.62.126:37855
I20260926 23:17:18.711489 24701 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:18.712224 24701 heartbeater.cc:511] Master 127.16.62.126:37855 requested a full tablet report, sending...
I20260926 23:17:18.713443 24217 ts_manager.cc:194] Registered new tserver with Master: 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007)
I20260926 23:17:18.714293 24217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:39737
I20260926 23:17:18.725020 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:17:18 dist-test-slave-vcv1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:17:18 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:18 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:19.223042 24562 heartbeater.cc:503] Master 127.16.62.126:37855 was elected leader, sending a full tablet report...
I20260926 23:17:19.716619 24701 heartbeater.cc:503] Master 127.16.62.126:37855 was elected leader, sending a full tablet report...
W20260926 23:17:19.832504 24709 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.067s user 0.006s sys 0.009s
W20260926 23:17:19.833643 24265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:18.760558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.16.62.126:37855)
0926 23:17:18.760780 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:18.760786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:18.760928 (+ 142us) server_negotiation.cc:407] Connection header received
0926 23:17:18.761081 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:18.761085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:18.761145 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:18.761225 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:18.762423 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.763101 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.764376 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:18.764790 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:18.764879 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.833348 (+1068469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43392: BlockingRecv error: recv got EOF from 127.0.0.1:43392 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 23:17:19 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:19.855665 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.837992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.16.62.126:37855)
0926 23:17:19.838192 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:19.838200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:19.838490 (+ 290us) server_negotiation.cc:407] Connection header received
0926 23:17:19.838710 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:19.838714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:19.838787 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:19.838887 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:19.840044 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.840834 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.842231 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.842708 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.842855 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.851073 (+ 8218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:19.851078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:19.851081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:19.851113 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:19.853536 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:19.854079 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:19.854082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:19.854084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:19.854192 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:19.854491 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:19.854494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:19.854496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:19.854741 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:19.854902 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:19.855260 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 23:17:19.855417 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 23:17:19.859624 24217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:17:19.866011 24217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:17:19.907374 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.879402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.16.62.67:46007)
0926 23:17:19.898825 (+ 19423us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:19.898836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:19.899486 (+ 650us) server_negotiation.cc:407] Connection header received
0926 23:17:19.899603 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:19.899613 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:19.899909 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:19.900765 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:19.900945 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.902183 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.905346 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.906326 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.906497 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.906712 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:19.906940 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 23:17:19.907091 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19242,"thread_start_us":19068,"threads_started":1}
I20260926 23:17:19.907471 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.878730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:46007 (local address 127.0.0.1:48272)
0926 23:17:19.899231 (+ 20501us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:19.899249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:19.899368 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:19.900169 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:19.900176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:19.900206 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:19.900571 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.900602 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.902313 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.902321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.905042 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.905057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.906524 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.906530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.906699 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.906751 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 23:17:19.906886 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 23:17:19.907056 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20373,"mutex_wait_us":36,"thread_start_us":19018,"threads_started":1}
I20260926 23:17:19.908336 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.877913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:33863 (local address 127.0.0.1:41624)
0926 23:17:19.898100 (+ 20187us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:19.898121 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:19.898246 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:19.900662 (+ 2416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:19.900667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:19.900689 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:19.901059 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.901070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.902860 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.902866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.905366 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.905376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.907389 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.907395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.907642 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.907677 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:17:19.907766 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 23:17:19.907892 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20007,"mutex_wait_us":413,"thread_start_us":21013,"threads_started":1}
I20260926 23:17:19.908844 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.877827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.16.62.65:42319)
0926 23:17:19.900462 (+ 22635us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:19.900469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:19.900488 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:17:19.900554 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:19.900558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:19.900859 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:19.901064 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:19.901651 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.903045 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.905773 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.907107 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.907215 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.907422 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:19.908440 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0926 23:17:19.908595 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22483,"thread_start_us":22351,"threads_started":1}
I20260926 23:17:19.909077 24723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.878041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.16.62.66:33863)
0926 23:17:19.900030 (+ 21989us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:19.900040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:19.900066 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:17:19.900184 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:19.900191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:19.900443 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:19.900671 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:19.901267 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.902714 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.905514 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.906932 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.907060 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.907249 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:19.908650 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0926 23:17:19.908808 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21838,"thread_start_us":21718,"threads_started":1}
I20260926 23:17:19.908360 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.877622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:42319 (local address 127.0.0.1:39490)
0926 23:17:19.897346 (+ 19724us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:19.897391 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:19.897568 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:19.901068 (+ 3500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:19.901073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:19.901102 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:19.901449 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.901460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.903183 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.903189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.905617 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.905628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.907530 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.907536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.907763 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.907789 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 23:17:19.907854 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 23:17:19.907945 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19472,"mutex_wait_us":73,"thread_start_us":19320,"threads_started":1}
I20260926 23:17:19.911195 24631 tablet_service.cc:1467] Processing CreateTablet for tablet f8670e79c2c4455eb2a3909a415ebf6b (DEFAULT_TABLE table=test-table [id=64d882f4af6d47f085130ea980e9e7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:19.911952 24353 tablet_service.cc:1467] Processing CreateTablet for tablet f8670e79c2c4455eb2a3909a415ebf6b (DEFAULT_TABLE table=test-table [id=64d882f4af6d47f085130ea980e9e7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:19.912621 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8670e79c2c4455eb2a3909a415ebf6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:19.912421 24492 tablet_service.cc:1467] Processing CreateTablet for tablet f8670e79c2c4455eb2a3909a415ebf6b (DEFAULT_TABLE table=test-table [id=64d882f4af6d47f085130ea980e9e7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:19.912979 24353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8670e79c2c4455eb2a3909a415ebf6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:19.913609 24492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8670e79c2c4455eb2a3909a415ebf6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:19.923507 24726 tablet_bootstrap.cc:492] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: Bootstrap starting.
I20260926 23:17:19.926342 24726 tablet_bootstrap.cc:654] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:19.927314 24726 log.cc:824] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:19.928946 24727 tablet_bootstrap.cc:492] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: Bootstrap starting.
I20260926 23:17:19.929793 24728 tablet_bootstrap.cc:492] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: Bootstrap starting.
I20260926 23:17:19.933897 24728 tablet_bootstrap.cc:654] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:19.935341 24728 log.cc:824] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:19.937160 24727 tablet_bootstrap.cc:654] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:19.938273 24727 log.cc:824] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:19.938510 24728 tablet_bootstrap.cc:492] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: No bootstrap required, opened a new log
I20260926 23:17:19.938804 24728 ts_tablet_manager.cc:1397] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 23:17:19.938869 24726 tablet_bootstrap.cc:492] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: No bootstrap required, opened a new log
I20260926 23:17:19.939076 24726 ts_tablet_manager.cc:1397] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260926 23:17:19.940933 24727 tablet_bootstrap.cc:492] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: No bootstrap required, opened a new log
I20260926 23:17:19.941180 24727 ts_tablet_manager.cc:1397] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260926 23:17:19.946151 24728 raft_consensus.cc:348] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:19.946480 24728 raft_consensus.cc:374] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:19.946596 24728 raft_consensus.cc:729] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d4b0ba6b9042c4a9e903d15e5f9032, State: Initialized, Role: FOLLOWER
I20260926 23:17:19.947145 24728 consensus_queue.cc:260] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:19.948872 24726 raft_consensus.cc:348] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:19.949287 24726 raft_consensus.cc:374] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:19.949415 24726 raft_consensus.cc:729] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab65987a9084ac2a96c0a977c774418, State: Initialized, Role: FOLLOWER
I20260926 23:17:19.949599 24728 ts_tablet_manager.cc:1428] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260926 23:17:19.950107 24726 consensus_queue.cc:260] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:19.951354 24727 raft_consensus.cc:348] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:19.951725 24726 ts_tablet_manager.cc:1428] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 23:17:19.951795 24727 raft_consensus.cc:374] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:19.951926 24727 raft_consensus.cc:729] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896629842d4048a7b51cbcd4af8f7252, State: Initialized, Role: FOLLOWER
I20260926 23:17:19.952937 24727 consensus_queue.cc:260] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:19.954458 24727 ts_tablet_manager.cc:1428] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260926 23:17:19.961001 24702 tablet.cc:2369] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:17:19.961882 24734 raft_consensus.cc:482] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:19.962095 24734 raft_consensus.cc:504] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:19.963424 24734 leader_election.cc:288] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319), a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863)
W20260926 23:17:19.966708 24563 tablet.cc:2369] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:17:19.969880 24723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.963972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:34543 (local address 127.16.62.66:33863)
0926 23:17:19.964159 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:19.964167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:19.964811 (+ 644us) server_negotiation.cc:407] Connection header received
0926 23:17:19.964991 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:19.964996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:19.965067 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:19.965170 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:19.965694 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.966368 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.968308 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.969019 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.969207 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.969281 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:19.969541 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 23:17:19.969655 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 23:17:19.969921 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.963631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:33135 (local address 127.16.62.65:42319)
0926 23:17:19.963827 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:19.963837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:19.964501 (+ 664us) server_negotiation.cc:407] Connection header received
0926 23:17:19.964720 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:19.964723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:19.964783 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:19.964874 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:19.965373 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.966070 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.968305 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.969045 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.969219 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.969298 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:19.969572 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 23:17:19.969687 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:17:19.970158 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.963626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:42319 (local address 127.16.62.67:33135)
0926 23:17:19.964399 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:19.964415 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:19.964579 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:19.964888 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:19.964892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:19.964918 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:19.965239 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.965247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.966189 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.966197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.968093 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.968104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.969304 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.969312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.969430 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.969451 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 23:17:19.969550 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 23:17:19.969666 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":60,"thread_start_us":533,"threads_started":1}
I20260926 23:17:19.970202 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:19.963854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:33863 (local address 127.16.62.67:34543)
0926 23:17:19.964707 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:19.964729 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:19.964868 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:19.965177 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:19.965181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:19.965213 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:19.965551 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.965559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.966501 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.966506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.968071 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:19.968085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:19.969247 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:19.969253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:19.969393 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:19.969415 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:17:19.969518 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 23:17:19.969638 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":25,"thread_start_us":529,"threads_started":1}
I20260926 23:17:19.971092 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8670e79c2c4455eb2a3909a415ebf6b" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab65987a9084ac2a96c0a977c774418" is_pre_election: true
I20260926 23:17:19.971158 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8670e79c2c4455eb2a3909a415ebf6b" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" is_pre_election: true
I20260926 23:17:19.971604 24516 raft_consensus.cc:2454] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 0.
I20260926 23:17:19.971604 24378 raft_consensus.cc:2454] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 0.
I20260926 23:17:19.972250 24584 leader_election.cc:302] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, 896629842d4048a7b51cbcd4af8f7252; no voters:
I20260926 23:17:19.972568 24734 raft_consensus.cc:2781] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:17:19.972695 24734 raft_consensus.cc:482] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:17:19.972815 24734 raft_consensus.cc:3037] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:19.974843 24734 raft_consensus.cc:504] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:19.975389 24734 leader_election.cc:288] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 election: Requested vote from peers 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319), a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863)
I20260926 23:17:19.975775 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8670e79c2c4455eb2a3909a415ebf6b" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab65987a9084ac2a96c0a977c774418"
I20260926 23:17:19.975798 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8670e79c2c4455eb2a3909a415ebf6b" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032"
I20260926 23:17:19.976001 24516 raft_consensus.cc:3037] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:19.976001 24378 raft_consensus.cc:3037] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:19.979147 24516 raft_consensus.cc:2454] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 1.
I20260926 23:17:19.979147 24378 raft_consensus.cc:2454] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 1.
I20260926 23:17:19.979514 24584 leader_election.cc:302] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, 896629842d4048a7b51cbcd4af8f7252; no voters:
I20260926 23:17:19.979851 24734 raft_consensus.cc:2781] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:19.980274 24734 raft_consensus.cc:686] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 1 LEADER]: Becoming Leader. State: Replica: 896629842d4048a7b51cbcd4af8f7252, State: Running, Role: LEADER
I20260926 23:17:19.980775 24734 consensus_queue.cc:237] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:19.985533 24217 catalog_manager.cc:5495] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67). New cstate: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } health_report { overall_health: HEALTHY } } }
I20260926 23:17:20.013538 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:20.008825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.16.62.67:46007)
0926 23:17:20.009007 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:20.009011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:20.009148 (+ 137us) server_negotiation.cc:407] Connection header received
0926 23:17:20.009323 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:20.009326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:20.009429 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:20.009548 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:20.009851 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.010502 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.011745 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.012114 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.012221 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:20.012569 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:20.013057 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:20.013152 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:20.013291 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 23:17:20.013347 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 23:17:20.021071 24378 raft_consensus.cc:1261] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Refusing update from remote peer 896629842d4048a7b51cbcd4af8f7252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:20.021075 24516 raft_consensus.cc:1261] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Refusing update from remote peer 896629842d4048a7b51cbcd4af8f7252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:20.022133 24737 consensus_queue.cc:1035] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:20.022475 24734 consensus_queue.cc:1035] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:20.047271 24216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:20.047536 24216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:20.051000 24216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d: Sending DeleteTablet for 3 replicas of tablet f8670e79c2c4455eb2a3909a415ebf6b
I20260926 23:17:20.052213 24353 tablet_service.cc:1514] Processing DeleteTablet for tablet f8670e79c2c4455eb2a3909a415ebf6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:20 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:39490
I20260926 23:17:20.052913 24492 tablet_service.cc:1514] Processing DeleteTablet for tablet f8670e79c2c4455eb2a3909a415ebf6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:20 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:41624
I20260926 23:17:20.052969 24631 tablet_service.cc:1514] Processing DeleteTablet for tablet f8670e79c2c4455eb2a3909a415ebf6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:20 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:48272
I20260926 23:17:20.053077 24750 tablet_replica.cc:330] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: stopping tablet replica
I20260926 23:17:20.053503 24750 raft_consensus.cc:2229] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:20.053980 24750 raft_consensus.cc:2258] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:20.054018 24216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:17:20.055130 24216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:17:20.062765 24491 tablet_service.cc:1467] Processing CreateTablet for tablet 9d335b3052784418b14496638da3a680 (DEFAULT_TABLE table=test-table [id=f1c63d47d6844aa3ad8c23c9d929e22e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:20.063019 24630 tablet_service.cc:1467] Processing CreateTablet for tablet 9d335b3052784418b14496638da3a680 (DEFAULT_TABLE table=test-table [id=f1c63d47d6844aa3ad8c23c9d929e22e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:20.063374 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d335b3052784418b14496638da3a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:20.063603 24630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d335b3052784418b14496638da3a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:20.064352 24752 tablet_replica.cc:330] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: stopping tablet replica
I20260926 23:17:20.064785 24752 raft_consensus.cc:2229] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:20.064782 24353 tablet_service.cc:1467] Processing CreateTablet for tablet 9d335b3052784418b14496638da3a680 (DEFAULT_TABLE table=test-table [id=f1c63d47d6844aa3ad8c23c9d929e22e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:20.065152 24353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d335b3052784418b14496638da3a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:20.065315 24752 raft_consensus.cc:2258] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:20.066231 24751 tablet_replica.cc:330] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: stopping tablet replica
I20260926 23:17:20.066659 24751 raft_consensus.cc:2229] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:17:20.067369 24751 raft_consensus.cc:2258] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:20.069149 24750 ts_tablet_manager.cc:1905] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:20.070511 24728 tablet_bootstrap.cc:492] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: Bootstrap starting.
I20260926 23:17:20.071429 24726 tablet_bootstrap.cc:492] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: Bootstrap starting.
I20260926 23:17:20.071995 24728 tablet_bootstrap.cc:654] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:20.072947 24726 tablet_bootstrap.cc:654] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:20.073696 24728 tablet_bootstrap.cc:492] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: No bootstrap required, opened a new log
I20260926 23:17:20.073832 24728 ts_tablet_manager.cc:1397] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 23:17:20.074674 24728 raft_consensus.cc:348] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.074869 24728 raft_consensus.cc:374] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:20.074934 24728 raft_consensus.cc:729] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d4b0ba6b9042c4a9e903d15e5f9032, State: Initialized, Role: FOLLOWER
I20260926 23:17:20.075354 24727 tablet_bootstrap.cc:492] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: Bootstrap starting.
I20260926 23:17:20.075317 24728 consensus_queue.cc:260] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.076404 24750 ts_tablet_manager.cc:1918] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:20.076598 24726 tablet_bootstrap.cc:492] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: No bootstrap required, opened a new log
I20260926 23:17:20.076646 24728 ts_tablet_manager.cc:1428] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 23:17:20.076753 24726 ts_tablet_manager.cc:1397] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 23:17:20.076921 24727 tablet_bootstrap.cc:654] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:20.077502 24726 raft_consensus.cc:348] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.077736 24726 raft_consensus.cc:374] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:20.077833 24726 raft_consensus.cc:729] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab65987a9084ac2a96c0a977c774418, State: Initialized, Role: FOLLOWER
I20260926 23:17:20.078245 24726 consensus_queue.cc:260] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.079655 24726 ts_tablet_manager.cc:1428] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 23:17:20.076606 24750 log.cc:1196] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/f8670e79c2c4455eb2a3909a415ebf6b
I20260926 23:17:20.082305 24750 ts_tablet_manager.cc:1939] T f8670e79c2c4455eb2a3909a415ebf6b P 3ab65987a9084ac2a96c0a977c774418: Deleting consensus metadata
I20260926 23:17:20.082839 24751 ts_tablet_manager.cc:1905] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:20.083581 24727 tablet_bootstrap.cc:492] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: No bootstrap required, opened a new log
I20260926 23:17:20.083729 24727 ts_tablet_manager.cc:1397] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260926 23:17:20.084470 24203 catalog_manager.cc:4841] TS 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319): tablet f8670e79c2c4455eb2a3909a415ebf6b (table test-table [id=64d882f4af6d47f085130ea980e9e7e7]) successfully deleted
I20260926 23:17:20.084604 24727 raft_consensus.cc:348] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.084833 24727 raft_consensus.cc:374] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:20.084949 24727 raft_consensus.cc:729] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896629842d4048a7b51cbcd4af8f7252, State: Initialized, Role: FOLLOWER
I20260926 23:17:20.085225 24727 consensus_queue.cc:260] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.085774 24727 ts_tablet_manager.cc:1428] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 23:17:20.088573 24752 ts_tablet_manager.cc:1905] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:20.089872 24751 ts_tablet_manager.cc:1918] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:20.090037 24751 log.cc:1196] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/f8670e79c2c4455eb2a3909a415ebf6b
I20260926 23:17:20.090515 24751 ts_tablet_manager.cc:1939] T f8670e79c2c4455eb2a3909a415ebf6b P 896629842d4048a7b51cbcd4af8f7252: Deleting consensus metadata
I20260926 23:17:20.091804 24202 catalog_manager.cc:4841] TS 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007): tablet f8670e79c2c4455eb2a3909a415ebf6b (table test-table [id=64d882f4af6d47f085130ea980e9e7e7]) successfully deleted
I20260926 23:17:20.095265 24752 ts_tablet_manager.cc:1918] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:20.095415 24752 log.cc:1196] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/f8670e79c2c4455eb2a3909a415ebf6b
I20260926 23:17:20.095949 24752 ts_tablet_manager.cc:1939] T f8670e79c2c4455eb2a3909a415ebf6b P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting consensus metadata
I20260926 23:17:20.097527 24204 catalog_manager.cc:4841] TS a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863): tablet f8670e79c2c4455eb2a3909a415ebf6b (table test-table [id=64d882f4af6d47f085130ea980e9e7e7]) successfully deleted
W20260926 23:17:20.182623 24424 tablet.cc:2369] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:17:20.223701 24746 raft_consensus.cc:482] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:20.223941 24746 raft_consensus.cc:504] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.224798 24746 leader_election.cc:288] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319), a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863)
I20260926 23:17:20.225291 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d335b3052784418b14496638da3a680" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab65987a9084ac2a96c0a977c774418" is_pre_election: true
I20260926 23:17:20.225550 24378 raft_consensus.cc:2454] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 0.
I20260926 23:17:20.225524 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d335b3052784418b14496638da3a680" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" is_pre_election: true
I20260926 23:17:20.225780 24516 raft_consensus.cc:2454] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 0.
I20260926 23:17:20.226009 24584 leader_election.cc:302] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, 896629842d4048a7b51cbcd4af8f7252; no voters:
I20260926 23:17:20.226362 24746 raft_consensus.cc:2781] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:17:20.226496 24746 raft_consensus.cc:482] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:17:20.226593 24746 raft_consensus.cc:3037] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:20.228886 24746 raft_consensus.cc:504] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.230012 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d335b3052784418b14496638da3a680" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab65987a9084ac2a96c0a977c774418"
I20260926 23:17:20.230016 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d335b3052784418b14496638da3a680" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032"
I20260926 23:17:20.230218 24516 raft_consensus.cc:3037] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:20.230211 24378 raft_consensus.cc:3037] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:20.233297 24378 raft_consensus.cc:2454] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 1.
I20260926 23:17:20.233304 24516 raft_consensus.cc:2454] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 1.
I20260926 23:17:20.233788 24584 leader_election.cc:302] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, 896629842d4048a7b51cbcd4af8f7252; no voters:
I20260926 23:17:20.234099 24734 raft_consensus.cc:2781] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:20.234138 24746 leader_election.cc:288] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 election: Requested vote from peers 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319), a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863)
I20260926 23:17:20.234261 24734 raft_consensus.cc:686] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 1 LEADER]: Becoming Leader. State: Replica: 896629842d4048a7b51cbcd4af8f7252, State: Running, Role: LEADER
I20260926 23:17:20.234525 24734 consensus_queue.cc:237] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.237470 24217 catalog_manager.cc:5495] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67). New cstate: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } health_report { overall_health: HEALTHY } } }
Sep 26 23:17:20 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464637, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.126@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:20.297610 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:20.282703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37855 (local address 127.0.0.1:43418)
0926 23:17:20.283334 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:20.283354 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:20.283507 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:20.283863 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:20.283867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:20.284563 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:20.284816 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:20.284825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.286287 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.286291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:20.287255 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:20.287266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.288293 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.288297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:20.288405 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:20.289085 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:20.289102 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:20.291389 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:20.295131 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:20.295140 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:20.295145 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:20.295558 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:20.295940 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:20.295943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:20.295945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:20.296102 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:20.296795 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:20.296800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:20.296941 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 23:17:20.297110 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 23:17:20.297308 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":304,"threads_started":1}
I20260926 23:17:20.297912 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:20.282542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.16.62.126:37855)
0926 23:17:20.282767 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:20.282771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:20.283450 (+ 679us) server_negotiation.cc:407] Connection header received
0926 23:17:20.283633 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:20.283638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:20.283716 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:20.283820 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:20.284948 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.286142 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.287393 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.288064 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.288248 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:20.291577 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:20.291584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:20.291589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:20.291626 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:20.294987 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:20.295672 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:20.295678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:20.295682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:20.295835 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:20.296285 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:20.296290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:20.296294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:20.296646 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:20.296812 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:20.297273 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 23:17:20.297492 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:17:20.302965 24217 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:43418:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 23:17:20.304080 24217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:17:20.310969 24353 tablet_service.cc:1467] Processing CreateTablet for tablet e2c1f3446e3748ecb69cd9d8f2d07c97 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59d826ae02644c27bc8bcd32e8f18a72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:17:20.310976 24491 tablet_service.cc:1467] Processing CreateTablet for tablet e2c1f3446e3748ecb69cd9d8f2d07c97 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59d826ae02644c27bc8bcd32e8f18a72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:17:20.311480 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c1f3446e3748ecb69cd9d8f2d07c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:20.311503 24353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c1f3446e3748ecb69cd9d8f2d07c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:20.315414 24630 tablet_service.cc:1467] Processing CreateTablet for tablet e2c1f3446e3748ecb69cd9d8f2d07c97 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59d826ae02644c27bc8bcd32e8f18a72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:17:20.316079 24630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c1f3446e3748ecb69cd9d8f2d07c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:20.318642 24728 tablet_bootstrap.cc:492] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: Bootstrap starting.
I20260926 23:17:20.318648 24726 tablet_bootstrap.cc:492] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: Bootstrap starting.
I20260926 23:17:20.320335 24728 tablet_bootstrap.cc:654] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:20.320335 24726 tablet_bootstrap.cc:654] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:20.324560 24727 tablet_bootstrap.cc:492] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: Bootstrap starting.
I20260926 23:17:20.326359 24727 tablet_bootstrap.cc:654] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:20.330885 24728 tablet_bootstrap.cc:492] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: No bootstrap required, opened a new log
I20260926 23:17:20.331415 24728 ts_tablet_manager.cc:1397] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20260926 23:17:20.332058 24727 tablet_bootstrap.cc:492] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: No bootstrap required, opened a new log
I20260926 23:17:20.332489 24727 ts_tablet_manager.cc:1397] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:17:20.332347 24728 raft_consensus.cc:348] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.332758 24728 raft_consensus.cc:374] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:20.332906 24726 tablet_bootstrap.cc:492] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: No bootstrap required, opened a new log
I20260926 23:17:20.332887 24728 raft_consensus.cc:729] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d4b0ba6b9042c4a9e903d15e5f9032, State: Initialized, Role: FOLLOWER
I20260926 23:17:20.333122 24726 ts_tablet_manager.cc:1397] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260926 23:17:20.333205 24728 consensus_queue.cc:260] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.333232 24727 raft_consensus.cc:348] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.333489 24727 raft_consensus.cc:374] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:20.333578 24727 raft_consensus.cc:729] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896629842d4048a7b51cbcd4af8f7252, State: Initialized, Role: FOLLOWER
I20260926 23:17:20.333832 24727 consensus_queue.cc:260] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.333753 24732 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.333894 24726 raft_consensus.cc:348] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.334122 24732 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:17:20.334110 24726 raft_consensus.cc:374] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:20.334182 24726 raft_consensus.cc:729] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab65987a9084ac2a96c0a977c774418, State: Initialized, Role: FOLLOWER
I20260926 23:17:20.334405 24727 ts_tablet_manager.cc:1428] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 23:17:20.334436 24726 consensus_queue.cc:260] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.334853 24734 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.335136 24734 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:17:20.335124 24726 ts_tablet_manager.cc:1428] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 23:17:20.334920 24733 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.335282 24733 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:17:20.336685 24728 ts_tablet_manager.cc:1428] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 23:17:20.443408 24734 raft_consensus.cc:482] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:20.443622 24734 raft_consensus.cc:504] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.444347 24734 leader_election.cc:288] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319), a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863)
I20260926 23:17:20.444859 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c1f3446e3748ecb69cd9d8f2d07c97" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab65987a9084ac2a96c0a977c774418" is_pre_election: true
I20260926 23:17:20.444931 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c1f3446e3748ecb69cd9d8f2d07c97" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" is_pre_election: true
I20260926 23:17:20.445103 24378 raft_consensus.cc:2454] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 0.
I20260926 23:17:20.445225 24516 raft_consensus.cc:2454] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 0.
I20260926 23:17:20.445622 24585 leader_election.cc:302] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 896629842d4048a7b51cbcd4af8f7252, a3d4b0ba6b9042c4a9e903d15e5f9032; no voters:
I20260926 23:17:20.445982 24734 raft_consensus.cc:2781] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:17:20.446094 24734 raft_consensus.cc:482] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:17:20.446264 24734 raft_consensus.cc:3037] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:20.448423 24734 raft_consensus.cc:504] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.449020 24734 leader_election.cc:288] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 election: Requested vote from peers 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319), a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863)
I20260926 23:17:20.449433 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c1f3446e3748ecb69cd9d8f2d07c97" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab65987a9084ac2a96c0a977c774418"
I20260926 23:17:20.449429 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c1f3446e3748ecb69cd9d8f2d07c97" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032"
I20260926 23:17:20.449657 24378 raft_consensus.cc:3037] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:20.449663 24516 raft_consensus.cc:3037] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:20.451834 24378 raft_consensus.cc:2454] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 1.
I20260926 23:17:20.452227 24584 leader_election.cc:302] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, 896629842d4048a7b51cbcd4af8f7252; no voters:
I20260926 23:17:20.452363 24516 raft_consensus.cc:2454] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 1.
I20260926 23:17:20.452484 24734 raft_consensus.cc:2781] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:20.452702 24734 raft_consensus.cc:686] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [term 1 LEADER]: Becoming Leader. State: Replica: 896629842d4048a7b51cbcd4af8f7252, State: Running, Role: LEADER
I20260926 23:17:20.453020 24734 consensus_queue.cc:237] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.454581 24737 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: TxnStatusTablet state changed. Reason: New leader 896629842d4048a7b51cbcd4af8f7252. Latest consensus state: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.454896 24737 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:17:20.456314 24216 catalog_manager.cc:5495] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67). New cstate: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } health_report { overall_health: HEALTHY } } }
I20260926 23:17:20.456638 24767 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 23:17:20.456859 24767 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 23:17:20.497802 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:20.492786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:46007 (local address 127.0.0.1:48288)
0926 23:17:20.493023 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:20.493044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:20.493217 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:20.493594 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:20.493598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:20.493611 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:17:20.493787 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:20.493795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.494809 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.494816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:20.495833 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:20.495843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.496664 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.496670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:20.496832 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:20.496881 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:17:20.497279 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:17:20.497306 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 23:17:20.497405 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 23:17:20.497541 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 23:17:20.497829 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:20.492836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.16.62.67:46007)
0926 23:17:20.493035 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:20.493043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:20.493121 (+ 78us) server_negotiation.cc:407] Connection header received
0926 23:17:20.493394 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:20.493406 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:20.493471 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:20.493567 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:20.493914 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.494686 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.495989 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.496505 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.496655 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:20.497063 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:20.497180 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:20.497275 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:20.497426 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 23:17:20.497503 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:17:20.501142 24378 raft_consensus.cc:1261] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Refusing update from remote peer 896629842d4048a7b51cbcd4af8f7252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:20.501192 24516 raft_consensus.cc:1261] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Refusing update from remote peer 896629842d4048a7b51cbcd4af8f7252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:20.501821 24734 consensus_queue.cc:1035] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:20.502111 24737 consensus_queue.cc:1035] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:20.504172 24733 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: TxnStatusTablet state changed. Reason: New leader 896629842d4048a7b51cbcd4af8f7252. Latest consensus state: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.504438 24733 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:17:20.506201 24737 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.506425 24737 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:17:20.507014 24740 mvcc.cc:205] Tried to move back new op lower bound from 7333743167487041536 to 7333743167297662976. Current Snapshot: MvccSnapshot[applied={T|T < 7333743167487041536 or (T in {7333743167487041536})}]
I20260926 23:17:20.508495 24733 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.508729 24733 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 3ab65987a9084ac2a96c0a977c774418: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:17:20.518385 24734 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.518659 24734 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P 896629842d4048a7b51cbcd4af8f7252: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:17:20.518534 24732 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: TxnStatusTablet state changed. Reason: New leader 896629842d4048a7b51cbcd4af8f7252. Latest consensus state: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.518814 24732 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:17:20.520434 24732 tablet_replica.cc:439] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } } }
I20260926 23:17:20.520659 24732 tablet_replica.cc:442] T e2c1f3446e3748ecb69cd9d8f2d07c97 P a3d4b0ba6b9042c4a9e903d15e5f9032: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:17:20.530226 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:20.523785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:32961 (local address 127.16.62.67:46007)
0926 23:17:20.523965 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:20.523973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:20.524601 (+ 628us) server_negotiation.cc:407] Connection header received
0926 23:17:20.524776 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:20.524780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:20.524832 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:20.524901 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:20.525510 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.526259 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.528410 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.528825 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.528996 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:20.529438 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:20.529544 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:20.529610 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:20.529822 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 23:17:20.529881 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 23:17:20.530385 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:20.523806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:46007 (local address 127.16.62.67:32961)
0926 23:17:20.524495 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:20.524519 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:20.524665 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:20.524948 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:20.524953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:20.524964 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:17:20.525361 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:20.525375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.526404 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.526409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:20.528277 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:20.528295 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.529014 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.529021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:20.529210 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:20.529284 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:17:20.529649 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:17:20.529660 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:17:20.529778 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 23:17:20.529948 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":342,"threads_started":1}
I20260926 23:17:20.539489 24378 raft_consensus.cc:1261] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Refusing update from remote peer 896629842d4048a7b51cbcd4af8f7252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:20.539790 24514 raft_consensus.cc:1261] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Refusing update from remote peer 896629842d4048a7b51cbcd4af8f7252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:20.540391 24734 consensus_queue.cc:1035] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:20.540686 24737 consensus_queue.cc:1035] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:20.607424 24216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:20.607668 24216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:20.610633 24216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d: Sending DeleteTablet for 3 replicas of tablet 9d335b3052784418b14496638da3a680
I20260926 23:17:20.611482 24353 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d335b3052784418b14496638da3a680 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:20 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:39490
I20260926 23:17:20.611488 24631 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d335b3052784418b14496638da3a680 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:20 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:48272
I20260926 23:17:20.611495 24491 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d335b3052784418b14496638da3a680 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:20 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:41624
I20260926 23:17:20.624360 24777 tablet_replica.cc:330] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: stopping tablet replica
I20260926 23:17:20.624586 24778 tablet_replica.cc:330] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: stopping tablet replica
I20260926 23:17:20.624723 24777 raft_consensus.cc:2229] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:20.624892 24778 raft_consensus.cc:2229] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:17:20.625031 24777 raft_consensus.cc:2258] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:20.625362 24778 raft_consensus.cc:2258] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:20.626235 24777 ts_tablet_manager.cc:1905] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:20.626403 24778 ts_tablet_manager.cc:1905] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:20.633451 24777 ts_tablet_manager.cc:1918] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:17:20.633623 24777 log.cc:1196] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/9d335b3052784418b14496638da3a680
I20260926 23:17:20.633637 24778 ts_tablet_manager.cc:1918] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:17:20.633790 24778 log.cc:1196] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/9d335b3052784418b14496638da3a680
I20260926 23:17:20.634183 24777 ts_tablet_manager.cc:1939] T 9d335b3052784418b14496638da3a680 P 3ab65987a9084ac2a96c0a977c774418: Deleting consensus metadata
I20260926 23:17:20.634306 24778 ts_tablet_manager.cc:1939] T 9d335b3052784418b14496638da3a680 P 896629842d4048a7b51cbcd4af8f7252: Deleting consensus metadata
I20260926 23:17:20.635272 24203 catalog_manager.cc:4841] TS 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319): tablet 9d335b3052784418b14496638da3a680 (table test-table [id=f1c63d47d6844aa3ad8c23c9d929e22e]) successfully deleted
I20260926 23:17:20.635346 24202 catalog_manager.cc:4841] TS 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007): tablet 9d335b3052784418b14496638da3a680 (table test-table [id=f1c63d47d6844aa3ad8c23c9d929e22e]) successfully deleted
I20260926 23:17:20.635797 24779 tablet_replica.cc:330] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: stopping tablet replica
I20260926 23:17:20.635986 24779 raft_consensus.cc:2229] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:20.636240 24779 raft_consensus.cc:2258] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:20.637199 24779 ts_tablet_manager.cc:1905] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:20.644361 24779 ts_tablet_manager.cc:1918] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:17:20.644488 24779 log.cc:1196] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/9d335b3052784418b14496638da3a680
I20260926 23:17:20.644932 24779 ts_tablet_manager.cc:1939] T 9d335b3052784418b14496638da3a680 P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting consensus metadata
I20260926 23:17:20.645979 24204 catalog_manager.cc:4841] TS a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863): tablet 9d335b3052784418b14496638da3a680 (table test-table [id=f1c63d47d6844aa3ad8c23c9d929e22e]) successfully deleted
Sep 26 23:17:20 dist-test-slave-vcv1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:17:20 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:20 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:17:20.681578 24265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:20.670252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.16.62.126:37855)
0926 23:17:20.670496 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:20.670505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:20.670871 (+ 366us) server_negotiation.cc:407] Connection header received
0926 23:17:20.671052 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:20.671056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:20.671118 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:20.671212 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:20.672900 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.673625 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.675447 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.675836 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.675926 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:20.681380 (+ 5454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43430: BlockingRecv error: recv got EOF from 127.0.0.1:43430 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 23:17:20 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM
I20260926 23:17:20.710397 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:20.689640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.16.62.126:37855)
0926 23:17:20.689863 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:20.689871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:20.696614 (+ 6743us) server_negotiation.cc:407] Connection header received
0926 23:17:20.696781 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:20.696787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:20.696856 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:20.696961 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:20.698092 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.698798 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.700029 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:20.700490 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:20.700619 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:20.704318 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:20.704323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:20.704326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:20.704356 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:20.707405 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:20.708205 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:20.708211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:20.708215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:20.708349 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:20.708756 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:20.708761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:20.708765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:20.709096 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:20.709331 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:20.709849 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 23:17:20.710048 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:17:20.718480 24216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:17:20.719208 24216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:17:20.726148 24353 tablet_service.cc:1467] Processing CreateTablet for tablet c67f3f7b604f4bd281aa26ef275e83f2 (DEFAULT_TABLE table=test-table [id=baf8a23b4fec4039b8f6330b94e405cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:20.726389 24631 tablet_service.cc:1467] Processing CreateTablet for tablet c67f3f7b604f4bd281aa26ef275e83f2 (DEFAULT_TABLE table=test-table [id=baf8a23b4fec4039b8f6330b94e405cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:20.726560 24353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67f3f7b604f4bd281aa26ef275e83f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:20.726763 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67f3f7b604f4bd281aa26ef275e83f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:20.732756 24491 tablet_service.cc:1467] Processing CreateTablet for tablet c67f3f7b604f4bd281aa26ef275e83f2 (DEFAULT_TABLE table=test-table [id=baf8a23b4fec4039b8f6330b94e405cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:20.733189 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67f3f7b604f4bd281aa26ef275e83f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:20.735556 24727 tablet_bootstrap.cc:492] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: Bootstrap starting.
I20260926 23:17:20.736964 24726 tablet_bootstrap.cc:492] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Bootstrap starting.
I20260926 23:17:20.737241 24727 tablet_bootstrap.cc:654] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:20.738698 24726 tablet_bootstrap.cc:654] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:20.739714 24728 tablet_bootstrap.cc:492] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: Bootstrap starting.
I20260926 23:17:20.741191 24728 tablet_bootstrap.cc:654] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:20.751147 24728 tablet_bootstrap.cc:492] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: No bootstrap required, opened a new log
I20260926 23:17:20.751353 24728 ts_tablet_manager.cc:1397] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260926 23:17:20.752169 24728 raft_consensus.cc:348] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.752540 24728 raft_consensus.cc:374] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:20.752672 24728 raft_consensus.cc:729] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d4b0ba6b9042c4a9e903d15e5f9032, State: Initialized, Role: FOLLOWER
I20260926 23:17:20.752960 24728 consensus_queue.cc:260] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.752332 24726 tablet_bootstrap.cc:492] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: No bootstrap required, opened a new log
I20260926 23:17:20.753611 24726 ts_tablet_manager.cc:1397] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260926 23:17:20.754124 24727 tablet_bootstrap.cc:492] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: No bootstrap required, opened a new log
I20260926 23:17:20.754325 24727 ts_tablet_manager.cc:1397] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.008s
I20260926 23:17:20.754439 24726 raft_consensus.cc:348] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.754739 24726 raft_consensus.cc:374] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:20.754868 24726 raft_consensus.cc:729] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab65987a9084ac2a96c0a977c774418, State: Initialized, Role: FOLLOWER
I20260926 23:17:20.755189 24726 consensus_queue.cc:260] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.755245 24727 raft_consensus.cc:348] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.755497 24727 raft_consensus.cc:374] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:20.755888 24728 ts_tablet_manager.cc:1428] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 23:17:20.755906 24726 ts_tablet_manager.cc:1428] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:17:20.756291 24727 raft_consensus.cc:729] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896629842d4048a7b51cbcd4af8f7252, State: Initialized, Role: FOLLOWER
I20260926 23:17:20.756610 24727 consensus_queue.cc:260] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:20.757323 24727 ts_tablet_manager.cc:1428] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 23:17:21.037067 24733 raft_consensus.cc:482] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:21.037269 24733 raft_consensus.cc:504] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.038684 24733 leader_election.cc:288] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863), 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007)
I20260926 23:17:21.052662 24734 raft_consensus.cc:482] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:21.052867 24734 raft_consensus.cc:504] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.052920 24732 raft_consensus.cc:482] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:21.053151 24732 raft_consensus.cc:504] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.053509 24734 leader_election.cc:288] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319), a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863)
I20260926 23:17:21.054246 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67f3f7b604f4bd281aa26ef275e83f2" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab65987a9084ac2a96c0a977c774418" is_pre_election: true
I20260926 23:17:21.054520 24378 raft_consensus.cc:2454] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 0.
I20260926 23:17:21.054896 24732 leader_election.cc:288] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319), 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007)
I20260926 23:17:21.055212 24584 leader_election.cc:302] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, 896629842d4048a7b51cbcd4af8f7252; no voters:
I20260926 23:17:21.055528 24734 raft_consensus.cc:2781] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:17:21.055656 24734 raft_consensus.cc:482] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:17:21.055753 24734 raft_consensus.cc:3037] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:21.059181 24734 raft_consensus.cc:504] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.060789 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67f3f7b604f4bd281aa26ef275e83f2" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab65987a9084ac2a96c0a977c774418"
I20260926 23:17:21.061015 24378 raft_consensus.cc:3037] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:21.062461 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.039027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:36067 (local address 127.16.62.67:46007)
0926 23:17:21.051581 (+ 12554us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.051590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.052160 (+ 570us) server_negotiation.cc:407] Connection header received
0926 23:17:21.052384 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.052388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.052441 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.052541 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:21.053115 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.054101 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.059066 (+ 4965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.060803 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.060968 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.061074 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.061809 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.062008 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12395,"thread_start_us":12284,"threads_started":1}
I20260926 23:17:21.063621 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.039094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:46007 (local address 127.16.62.65:36067)
0926 23:17:21.052036 (+ 12942us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:21.052049 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:21.052244 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:21.052553 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:21.052557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:21.052578 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:21.052951 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.052965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.054276 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.054280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.056999 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.057019 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.061279 (+ 4260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.061287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.061544 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.061611 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 23:17:21.062852 (+ 1241us) client_negotiation.cc:240] Negotiation successful
0926 23:17:21.063061 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12845,"mutex_wait_us":1,"spinlock_wait_cycles":2688,"thread_start_us":11860,"threads_started":1}
I20260926 23:17:21.064746 24378 raft_consensus.cc:2454] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 1.
I20260926 23:17:21.064589 24656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67f3f7b604f4bd281aa26ef275e83f2" candidate_uuid: "3ab65987a9084ac2a96c0a977c774418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896629842d4048a7b51cbcd4af8f7252" is_pre_election: true
I20260926 23:17:21.065039 24656 raft_consensus.cc:2379] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3ab65987a9084ac2a96c0a977c774418 in current term 1: Already voted for candidate 896629842d4048a7b51cbcd4af8f7252 in this term.
I20260926 23:17:21.065224 24584 leader_election.cc:302] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, 896629842d4048a7b51cbcd4af8f7252; no voters:
I20260926 23:17:21.065616 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67f3f7b604f4bd281aa26ef275e83f2" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" is_pre_election: true
I20260926 23:17:21.066011 24514 raft_consensus.cc:2454] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 0.
I20260926 23:17:21.066483 24746 raft_consensus.cc:2781] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:21.066555 24734 leader_election.cc:288] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [CANDIDATE]: Term 1 election: Requested vote from peers 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319), a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863)
I20260926 23:17:21.066625 24746 raft_consensus.cc:686] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 1 LEADER]: Becoming Leader. State: Replica: 896629842d4048a7b51cbcd4af8f7252, State: Running, Role: LEADER
I20260926 23:17:21.066924 24746 consensus_queue.cc:237] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.067411 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.039048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:33863 (local address 127.16.62.65:44541)
0926 23:17:21.051289 (+ 12241us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:21.051316 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:21.051467 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:21.051988 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:21.051992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:21.052017 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:21.052381 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.052390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.054180 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.054188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.058827 (+ 4639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.058845 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.066300 (+ 7455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.066309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.066547 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.066599 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 23:17:21.066691 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:17:21.066887 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12059,"mutex_wait_us":1956,"spinlock_wait_cycles":33152,"thread_start_us":12728,"threads_started":1}
I20260926 23:17:21.068327 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.039063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:44541 (local address 127.16.62.66:33863)
0926 23:17:21.051715 (+ 12652us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.051722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.051738 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:17:21.051803 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.051807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.051866 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.051973 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:21.052663 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.054011 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.064276 (+ 10265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.065722 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.066030 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.066136 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.067614 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.067826 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12523,"thread_start_us":12486,"threads_started":1}
I20260926 23:17:21.069598 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67f3f7b604f4bd281aa26ef275e83f2" candidate_uuid: "3ab65987a9084ac2a96c0a977c774418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" is_pre_election: true
I20260926 23:17:21.069864 24514 raft_consensus.cc:2454] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab65987a9084ac2a96c0a977c774418 in term 0.
I20260926 23:17:21.070358 24307 leader_election.cc:302] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, a3d4b0ba6b9042c4a9e903d15e5f9032; no voters: 896629842d4048a7b51cbcd4af8f7252
I20260926 23:17:21.070842 24733 raft_consensus.cc:2741] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260926 23:17:21.075304 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67f3f7b604f4bd281aa26ef275e83f2" candidate_uuid: "896629842d4048a7b51cbcd4af8f7252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032"
I20260926 23:17:21.075500 24514 raft_consensus.cc:3037] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:21.075181 24216 catalog_manager.cc:5495] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67). New cstate: current_term: 1 leader_uuid: "896629842d4048a7b51cbcd4af8f7252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } health_report { overall_health: HEALTHY } } }
I20260926 23:17:21.078748 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.055400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:42319 (local address 127.16.62.66:37851)
0926 23:17:21.068954 (+ 13554us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:21.068975 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:21.069123 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:21.069779 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:21.069786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:21.069818 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:21.070268 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.070276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.071856 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.071864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.074464 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.074478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.077902 (+ 3424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.077911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.078105 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.078140 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:17:21.078232 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:17:21.078369 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13390,"thread_start_us":18132,"threads_started":1}
I20260926 23:17:21.079388 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.056298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:37851 (local address 127.16.62.65:42319)
0926 23:17:21.065239 (+ 8941us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.065245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.069236 (+ 3991us) server_negotiation.cc:407] Connection header received
0926 23:17:21.069301 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.069308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.069387 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.069481 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:21.070439 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.071652 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.074642 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.075844 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.076005 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.076065 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.078947 (+ 2882us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.079084 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8815,"thread_start_us":12545,"threads_started":1}
I20260926 23:17:21.079510 24514 raft_consensus.cc:2454] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896629842d4048a7b51cbcd4af8f7252 in term 1.
I20260926 23:17:21.080052 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67f3f7b604f4bd281aa26ef275e83f2" candidate_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab65987a9084ac2a96c0a977c774418" is_pre_election: true
I20260926 23:17:21.080370 24378 raft_consensus.cc:2379] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3d4b0ba6b9042c4a9e903d15e5f9032 in current term 1: Already voted for candidate 896629842d4048a7b51cbcd4af8f7252 in this term.
I20260926 23:17:21.080740 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.059994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:45777 (local address 127.16.62.67:46007)
0926 23:17:21.062800 (+ 2806us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.062808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.073934 (+ 11126us) server_negotiation.cc:407] Connection header received
0926 23:17:21.074165 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.074170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.074241 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.074298 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:21.075090 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.076101 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.078637 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.079834 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.079997 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.080070 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.080382 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.080464 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2617,"thread_start_us":14080,"threads_started":1}
I20260926 23:17:21.080785 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.055784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:46007 (local address 127.16.62.66:45777)
0926 23:17:21.073846 (+ 18062us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:21.073865 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:21.074012 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:21.074469 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:21.074473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:21.074492 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:21.074909 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.074920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.076314 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.076319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.078454 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.078474 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.080048 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.080054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.080238 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.080271 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 23:17:21.080367 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:17:21.080479 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17952,"thread_start_us":13106,"threads_started":1}
I20260926 23:17:21.081357 24656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67f3f7b604f4bd281aa26ef275e83f2" candidate_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896629842d4048a7b51cbcd4af8f7252" is_pre_election: true
I20260926 23:17:21.081900 24444 leader_election.cc:302] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3d4b0ba6b9042c4a9e903d15e5f9032; no voters: 3ab65987a9084ac2a96c0a977c774418, 896629842d4048a7b51cbcd4af8f7252
I20260926 23:17:21.082311 24732 raft_consensus.cc:2726] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 23:17:21.111467 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.104794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.16.62.67:46007)
0926 23:17:21.105435 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.105440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.105454 (+ 14us) server_negotiation.cc:407] Connection header received
0926 23:17:21.105507 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.105511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.105568 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.105637 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:21.106040 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.107196 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.109117 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.109740 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.109868 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.110332 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:21.110461 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:21.110568 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.111022 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.111122 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540}
I20260926 23:17:21.114598 24378 raft_consensus.cc:1261] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Refusing update from remote peer 896629842d4048a7b51cbcd4af8f7252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:21.114854 24514 raft_consensus.cc:1261] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Refusing update from remote peer 896629842d4048a7b51cbcd4af8f7252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:21.115290 24737 consensus_queue.cc:1035] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:21.115535 24746 consensus_queue.cc:1035] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:21.130098 24216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:21.130470 24216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:21.134573 24216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d: Sending DeleteTablet for 3 replicas of tablet c67f3f7b604f4bd281aa26ef275e83f2
I20260926 23:17:21.135529 24353 tablet_service.cc:1514] Processing DeleteTablet for tablet c67f3f7b604f4bd281aa26ef275e83f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:21 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:39490
I20260926 23:17:21.135959 24631 tablet_service.cc:1514] Processing DeleteTablet for tablet c67f3f7b604f4bd281aa26ef275e83f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:21 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:48272
I20260926 23:17:21.136191 24812 tablet_replica.cc:330] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: stopping tablet replica
I20260926 23:17:21.136343 24491 tablet_service.cc:1514] Processing DeleteTablet for tablet c67f3f7b604f4bd281aa26ef275e83f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:21 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:41624
I20260926 23:17:21.136471 24813 tablet_replica.cc:330] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: stopping tablet replica
I20260926 23:17:21.136796 24779 tablet_replica.cc:330] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: stopping tablet replica
I20260926 23:17:21.137040 24779 raft_consensus.cc:2229] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:21.137247 24779 raft_consensus.cc:2258] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:21.137187 24216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:17:21.137876 24216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:17:21.138207 24779 ts_tablet_manager.cc:1905] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:21.139168 24813 raft_consensus.cc:2229] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:17:21.139560 24813 raft_consensus.cc:2258] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:21.140596 24813 ts_tablet_manager.cc:1905] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:21.145614 24779 ts_tablet_manager.cc:1918] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:21.145825 24779 log.cc:1196] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/c67f3f7b604f4bd281aa26ef275e83f2
I20260926 23:17:21.146484 24779 ts_tablet_manager.cc:1939] T c67f3f7b604f4bd281aa26ef275e83f2 P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting consensus metadata
I20260926 23:17:21.148157 24813 ts_tablet_manager.cc:1918] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:21.148299 24813 log.cc:1196] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/c67f3f7b604f4bd281aa26ef275e83f2
I20260926 23:17:21.148660 24813 ts_tablet_manager.cc:1939] T c67f3f7b604f4bd281aa26ef275e83f2 P 896629842d4048a7b51cbcd4af8f7252: Deleting consensus metadata
I20260926 23:17:21.149432 24204 catalog_manager.cc:4841] TS a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863): tablet c67f3f7b604f4bd281aa26ef275e83f2 (table test-table [id=baf8a23b4fec4039b8f6330b94e405cf]) successfully deleted
I20260926 23:17:21.149734 24202 catalog_manager.cc:4841] TS 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007): tablet c67f3f7b604f4bd281aa26ef275e83f2 (table test-table [id=baf8a23b4fec4039b8f6330b94e405cf]) successfully deleted
I20260926 23:17:21.150346 24491 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb2bb86f393427c8696de9553e02a0b (DEFAULT_TABLE table=test-table [id=f633744c818447feb14b8fd6e15faf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:21.150434 24631 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb2bb86f393427c8696de9553e02a0b (DEFAULT_TABLE table=test-table [id=f633744c818447feb14b8fd6e15faf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:21.150483 24353 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb2bb86f393427c8696de9553e02a0b (DEFAULT_TABLE table=test-table [id=f633744c818447feb14b8fd6e15faf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:21.150727 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb2bb86f393427c8696de9553e02a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:21.150800 24631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb2bb86f393427c8696de9553e02a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:21.150861 24353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb2bb86f393427c8696de9553e02a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:21.154840 24728 tablet_bootstrap.cc:492] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: Bootstrap starting.
I20260926 23:17:21.156445 24728 tablet_bootstrap.cc:654] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:21.157222 24726 tablet_bootstrap.cc:492] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: Bootstrap starting.
I20260926 23:17:21.158469 24726 tablet_bootstrap.cc:654] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:21.159031 24727 tablet_bootstrap.cc:492] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: Bootstrap starting.
I20260926 23:17:21.160080 24728 tablet_bootstrap.cc:492] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: No bootstrap required, opened a new log
I20260926 23:17:21.160290 24728 ts_tablet_manager.cc:1397] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260926 23:17:21.160984 24727 tablet_bootstrap.cc:654] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:21.161198 24728 raft_consensus.cc:348] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.161438 24728 raft_consensus.cc:374] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:21.161511 24728 raft_consensus.cc:729] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d4b0ba6b9042c4a9e903d15e5f9032, State: Initialized, Role: FOLLOWER
I20260926 23:17:21.161741 24728 consensus_queue.cc:260] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.162236 24728 ts_tablet_manager.cc:1428] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 23:17:21.162897 24812 raft_consensus.cc:2229] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:21.164189 24726 tablet_bootstrap.cc:492] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: No bootstrap required, opened a new log
I20260926 23:17:21.164357 24726 ts_tablet_manager.cc:1397] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260926 23:17:21.164940 24726 raft_consensus.cc:348] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.165144 24726 raft_consensus.cc:374] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:21.165222 24726 raft_consensus.cc:729] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab65987a9084ac2a96c0a977c774418, State: Initialized, Role: FOLLOWER
I20260926 23:17:21.165448 24726 consensus_queue.cc:260] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.166046 24726 ts_tablet_manager.cc:1428] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 23:17:21.167675 24733 raft_consensus.cc:482] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:21.167837 24733 raft_consensus.cc:504] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.168474 24733 leader_election.cc:288] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863), 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007)
I20260926 23:17:21.169008 24656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb2bb86f393427c8696de9553e02a0b" candidate_uuid: "3ab65987a9084ac2a96c0a977c774418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896629842d4048a7b51cbcd4af8f7252" is_pre_election: true
I20260926 23:17:21.169178 24812 pending_rounds.cc:62] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Trying to abort 2 pending ops.
I20260926 23:17:21.169245 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb2bb86f393427c8696de9553e02a0b" candidate_uuid: "3ab65987a9084ac2a96c0a977c774418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" is_pre_election: true
I20260926 23:17:21.169494 24514 raft_consensus.cc:2454] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab65987a9084ac2a96c0a977c774418 in term 0.
I20260926 23:17:21.169342 24812 pending_rounds.cc:68] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333743170000064512 op_type: WRITE_OP write_request { tablet_id: "c67f3f7b604f4bd281aa26ef275e83f2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\255\247\341\325\006\"8\n\ntest-admin\022*\n baf8a23b4fec4039b8f6330b94e405cf\020\001\030\001 \001(\001" signature: "<redacted>"")\305DW\220OeUQ\253}\037M\323{ZJ\210b\341\240\2570,zo\266D S\027\245\221ih\265\355\366\272\2351z\370(\027\264\355\323\"\272rb\"\344\\X\312\214~b\232\007u\035" signing_key_seq_num: 0 } } request_id { client_id: "0cf2e6e799e44e3a93dbfcda9bbff53e" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 23:17:21.169968 24307 leader_election.cc:302] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, a3d4b0ba6b9042c4a9e903d15e5f9032; no voters:
W20260926 23:17:21.170156 24305 leader_election.cc:341] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007): Illegal state: must be running to vote when last-logged opid is not known
I20260926 23:17:21.170308 24733 raft_consensus.cc:2781] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:17:21.170420 24812 pending_rounds.cc:68] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333743169812492288 op_type: NO_OP noop_request { }
I20260926 23:17:21.170475 24733 raft_consensus.cc:482] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:17:21.170553 24812 raft_consensus.cc:2866] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 23:17:21.170600 24733 raft_consensus.cc:3037] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:21.170697 24812 raft_consensus.cc:2258] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:21.172420 24727 tablet_bootstrap.cc:492] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: No bootstrap required, opened a new log
I20260926 23:17:21.172586 24727 ts_tablet_manager.cc:1397] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20260926 23:17:21.172783 24812 ts_tablet_manager.cc:1905] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:21.173115 24733 raft_consensus.cc:504] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.173342 24727 raft_consensus.cc:348] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.173581 24727 raft_consensus.cc:374] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:21.173728 24733 leader_election.cc:288] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [CANDIDATE]: Term 1 election: Requested vote from peers a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863), 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007)
I20260926 23:17:21.174178 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb2bb86f393427c8696de9553e02a0b" candidate_uuid: "3ab65987a9084ac2a96c0a977c774418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032"
I20260926 23:17:21.174386 24514 raft_consensus.cc:3037] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:21.174168 24656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb2bb86f393427c8696de9553e02a0b" candidate_uuid: "3ab65987a9084ac2a96c0a977c774418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896629842d4048a7b51cbcd4af8f7252"
I20260926 23:17:21.176285 24727 raft_consensus.cc:729] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896629842d4048a7b51cbcd4af8f7252, State: Initialized, Role: FOLLOWER
I20260926 23:17:21.176575 24727 consensus_queue.cc:260] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.176851 24656 raft_consensus.cc:3037] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:21.177183 24727 ts_tablet_manager.cc:1428] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 23:17:21.177259 24514 raft_consensus.cc:2454] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab65987a9084ac2a96c0a977c774418 in term 1.
I20260926 23:17:21.177716 24307 leader_election.cc:302] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab65987a9084ac2a96c0a977c774418, a3d4b0ba6b9042c4a9e903d15e5f9032; no voters:
I20260926 23:17:21.178013 24733 raft_consensus.cc:2781] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:21.178416 24733 raft_consensus.cc:686] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 1 LEADER]: Becoming Leader. State: Replica: 3ab65987a9084ac2a96c0a977c774418, State: Running, Role: LEADER
I20260926 23:17:21.178853 24733 consensus_queue.cc:237] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } }
I20260926 23:17:21.179481 24656 raft_consensus.cc:2454] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab65987a9084ac2a96c0a977c774418 in term 1.
I20260926 23:17:21.179699 24812 ts_tablet_manager.cc:1918] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:21.179823 24812 log.cc:1196] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/c67f3f7b604f4bd281aa26ef275e83f2
I20260926 23:17:21.180330 24812 ts_tablet_manager.cc:1939] T c67f3f7b604f4bd281aa26ef275e83f2 P 3ab65987a9084ac2a96c0a977c774418: Deleting consensus metadata
I20260926 23:17:21.181627 24203 catalog_manager.cc:4841] TS 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319): tablet c67f3f7b604f4bd281aa26ef275e83f2 (table test-table [id=baf8a23b4fec4039b8f6330b94e405cf]) successfully deleted
I20260926 23:17:21.186167 24216 catalog_manager.cc:5495] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "3ab65987a9084ac2a96c0a977c774418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab65987a9084ac2a96c0a977c774418" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 42319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 } health_report { overall_health: UNKNOWN } } }
I20260926 23:17:21.209537 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.204120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.16.62.65:42319)
0926 23:17:21.204392 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.204398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.204623 (+ 225us) server_negotiation.cc:407] Connection header received
0926 23:17:21.204843 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.204848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.204969 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.205136 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:21.205526 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.206217 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.207472 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.207870 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.207995 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.208448 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:21.208980 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:21.209131 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.209207 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.209284 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:17:21.224479 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.218893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:54991 (local address 127.16.62.67:46007)
0926 23:17:21.219057 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.219063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.219660 (+ 597us) server_negotiation.cc:407] Connection header received
0926 23:17:21.219846 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.219849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.219906 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.219974 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:21.220479 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.221135 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.222974 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.223368 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.223453 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.223907 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:21.223992 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:21.224055 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.224235 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.224293 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:17:21.224659 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.218833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:46007 (local address 127.16.62.65:54991)
0926 23:17:21.219553 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:21.219570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:21.219701 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:21.220006 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:21.220010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:21.220021 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:17:21.220339 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.220349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.221277 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.221282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.222855 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.222867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.223519 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.223525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.223703 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.223762 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:17:21.224076 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:17:21.224086 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:17:21.224177 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 23:17:21.224312 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":393,"threads_started":1}
I20260926 23:17:21.233003 24514 raft_consensus.cc:1261] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Refusing update from remote peer 3ab65987a9084ac2a96c0a977c774418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:21.233042 24656 raft_consensus.cc:1261] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Refusing update from remote peer 3ab65987a9084ac2a96c0a977c774418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:21.233587 24817 consensus_queue.cc:1035] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d4b0ba6b9042c4a9e903d15e5f9032" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:21.233995 24733 consensus_queue.cc:1035] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "896629842d4048a7b51cbcd4af8f7252" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:21.238384 24806 mvcc.cc:205] Tried to move back new op lower bound from 7333743170483945472 to 7333743170274631680. Current Snapshot: MvccSnapshot[applied={T|T < 7333743170483945472}]
I20260926 23:17:21.239811 24807 mvcc.cc:205] Tried to move back new op lower bound from 7333743170483945472 to 7333743170274631680. Current Snapshot: MvccSnapshot[applied={T|T < 7333743170483945472}]
I20260926 23:17:21.258718 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.253439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:37519 (local address 127.16.62.65:42319)
0926 23:17:21.253768 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.253773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.254048 (+ 275us) server_negotiation.cc:407] Connection header received
0926 23:17:21.254155 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.254158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.254212 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.254282 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:21.254638 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.255530 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.257134 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.257547 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.257674 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.258028 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:21.258128 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:21.258222 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.258364 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.258434 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260926 23:17:21.258808 24827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.253437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:42319 (local address 127.16.62.67:37519)
0926 23:17:21.253954 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:21.253969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:21.254075 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:21.254305 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:21.254308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:21.254316 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:17:21.254531 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.254538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.255652 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.255660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.257002 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:21.257014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.257680 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.257686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:21.257831 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.257862 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:17:21.258222 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:17:21.258244 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:17:21.258343 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 23:17:21.258476 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":333,"threads_started":1}
I20260926 23:17:21.288915 24215 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:21.289203 24215 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:21.292250 24215 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eb64b9e04f04017a4c3fa465233bc4d: Sending DeleteTablet for 3 replicas of tablet 1cb2bb86f393427c8696de9553e02a0b
I20260926 23:17:21.292917 24353 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb2bb86f393427c8696de9553e02a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:21 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:39490
I20260926 23:17:21.293012 24491 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb2bb86f393427c8696de9553e02a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:21 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:41624
I20260926 23:17:21.293188 24631 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb2bb86f393427c8696de9553e02a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:21 UTC) from {username='oryx', principal='oryx/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:48272
I20260926 23:17:21.293357 24812 tablet_replica.cc:330] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: stopping tablet replica
I20260926 23:17:21.293491 24813 tablet_replica.cc:330] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: stopping tablet replica
I20260926 23:17:21.293581 24812 raft_consensus.cc:2229] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:17:21.293699 24813 raft_consensus.cc:2229] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:21.293879 24813 raft_consensus.cc:2258] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:21.294075 24812 raft_consensus.cc:2258] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:21.294164 24779 tablet_replica.cc:330] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: stopping tablet replica
I20260926 23:17:21.294330 24779 raft_consensus.cc:2229] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:21.294551 24779 raft_consensus.cc:2258] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:21.295130 24812 ts_tablet_manager.cc:1905] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:21.295605 24779 ts_tablet_manager.cc:1905] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:21.295724 24813 ts_tablet_manager.cc:1905] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:21.298312 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 24288
I20260926 23:17:21.302153 24813 ts_tablet_manager.cc:1918] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:17:21.302152 24812 ts_tablet_manager.cc:1918] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:17:21.302289 24813 log.cc:1196] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/1cb2bb86f393427c8696de9553e02a0b
I20260926 23:17:21.302438 24812 log.cc:1196] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/1cb2bb86f393427c8696de9553e02a0b
I20260926 23:17:21.302659 24813 ts_tablet_manager.cc:1939] T 1cb2bb86f393427c8696de9553e02a0b P 896629842d4048a7b51cbcd4af8f7252: Deleting consensus metadata
I20260926 23:17:21.302975 24812 ts_tablet_manager.cc:1939] T 1cb2bb86f393427c8696de9553e02a0b P 3ab65987a9084ac2a96c0a977c774418: Deleting consensus metadata
I20260926 23:17:21.303633 24202 catalog_manager.cc:4841] TS 896629842d4048a7b51cbcd4af8f7252 (127.16.62.67:46007): tablet 1cb2bb86f393427c8696de9553e02a0b (table test-table [id=f633744c818447feb14b8fd6e15faf3b]) successfully deleted
I20260926 23:17:21.304075 24203 catalog_manager.cc:4841] TS 3ab65987a9084ac2a96c0a977c774418 (127.16.62.65:42319): tablet 1cb2bb86f393427c8696de9553e02a0b (table test-table [id=f633744c818447feb14b8fd6e15faf3b]) successfully deleted
I20260926 23:17:21.307770 24779 ts_tablet_manager.cc:1918] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:17:21.307937 24779 log.cc:1196] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/1cb2bb86f393427c8696de9553e02a0b
I20260926 23:17:21.308427 24779 ts_tablet_manager.cc:1939] T 1cb2bb86f393427c8696de9553e02a0b P a3d4b0ba6b9042c4a9e903d15e5f9032: Deleting consensus metadata
I20260926 23:17:21.309418 24204 catalog_manager.cc:4841] TS a3d4b0ba6b9042c4a9e903d15e5f9032 (127.16.62.66:33863): tablet 1cb2bb86f393427c8696de9553e02a0b (table test-table [id=f633744c818447feb14b8fd6e15faf3b]) successfully deleted
Sep 26 23:17:21 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.65@KRBTEST.COM
I20260926 23:17:21.324842 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.300248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.16.62.65:42319)
0926 23:17:21.301459 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.301465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.308706 (+ 7241us) server_negotiation.cc:407] Connection header received
0926 23:17:21.308881 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.308885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.308966 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.309070 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:21.310585 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.311346 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.313146 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.313547 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.313650 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.318226 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:21.318240 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:21.318259 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:21.318305 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:21.322140 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:21.322809 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:21.322819 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:21.322823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:21.322961 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:21.323292 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:21.323297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:21.323300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:21.323799 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:21.324110 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.324291 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.324504 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097}
I20260926 23:17:21.325605 24418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:21.431696 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 24288
I20260926 23:17:21.451673 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 24427
Sep 26 23:17:21 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.66@KRBTEST.COM
I20260926 23:17:21.466609 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.452455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.16.62.66:33863)
0926 23:17:21.452616 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.452620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.452713 (+ 93us) server_negotiation.cc:407] Connection header received
0926 23:17:21.452853 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.452856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.452931 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.453011 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:21.454147 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.454851 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.456553 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.456979 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.457104 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.460955 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:21.460966 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:21.460979 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:21.461033 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:21.464023 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:21.464767 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:21.464773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:21.464776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:21.464932 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:21.465259 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:21.465262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:21.465264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:21.465670 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:21.465883 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.466138 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.466304 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 23:17:21.467365 24557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:21.565543 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 24427
I20260926 23:17:21.583491 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 24566
Sep 26 23:17:21 dist-test-slave-vcv1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.62.67@KRBTEST.COM
I20260926 23:17:21.597177 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:21.584253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.16.62.67:46007)
0926 23:17:21.584431 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:21.584435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:21.584453 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:17:21.584580 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:21.584584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:21.584643 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:21.584744 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:21.585924 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.586595 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.587746 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:21.588079 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:21.588223 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:21.592347 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:21.592357 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:21.592373 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:21.592400 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:21.594934 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:21.595524 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:21.595531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:21.595533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:21.595645 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:21.595937 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:21.595940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:21.595942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:21.596294 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:21.596538 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:21.596686 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 23:17:21.596833 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:17:21.597821 24696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:21.743142 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 24566
I20260926 23:17:21.768512 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 24185
I20260926 23:17:21.769676 24257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:21.885438 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 24185
2026-09-26T23:17:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6491 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:17:21 dist-test-slave-vcv1 krb5kdc[24838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:17:21 dist-test-slave-vcv1 krb5kdc[24838](info): set up 2 sockets
Sep 26 23:17:21 dist-test-slave-vcv1 krb5kdc[24838](info): commencing operation
krb5kdc: starting...
W20260926 23:17:22.973848 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:17:23 dist-test-slave-vcv1 krb5kdc[24838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:17:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:17:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:17:23.184095 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:33387
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:23.491972 24851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:23.492318 24851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:23.492478 24851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:23.499428 24851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:23.499571 24851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:23.499679 24851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:23.499786 24851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:23.499858 24851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:23.509163 24851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:33387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:33387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:23.510381 24851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:23.511615 24851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:23.522948 24859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:23.523041 24856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:23.523015 24857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:23.523715 24851 server_base.cc:1034] running on GCE node
I20260926 23:17:23.524519 24851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:23.525866 24851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:23.527107 24851 hybrid_clock.cc:648] HybridClock initialized: now 1790464643527049 us; error 44 us; skew 500 ppm
I20260926 23:17:23.527441 24851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T23:17:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1702 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:17:23 dist-test-slave-vcv1 krb5kdc[24865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:17:23 dist-test-slave-vcv1 krb5kdc[24865](info): set up 2 sockets
Sep 26 23:17:23 dist-test-slave-vcv1 krb5kdc[24865](info): commencing operation
krb5kdc: starting...
W20260926 23:17:24.682243 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:17:24 dist-test-slave-vcv1 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:17:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:17:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:17:24.888696 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:35669
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:35669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:25.200385 24878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:25.200656 24878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:25.200850 24878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:25.207832 24878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:25.207959 24878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:25.208046 24878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:25.208164 24878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:25.208271 24878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:25.217746 24878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:35669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:35669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:25.218969 24878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:25.220242 24878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:25.231451 24883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:25.231484 24884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:25.231772 24886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:25.232187 24878 server_base.cc:1034] running on GCE node
I20260926 23:17:25.232909 24878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:25.234205 24878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:25.235432 24878 hybrid_clock.cc:648] HybridClock initialized: now 1790464645235348 us; error 71 us; skew 500 ppm
I20260926 23:17:25.235754 24878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:25 dist-test-slave-vcv1 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464645, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:25.253847 24878 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:17:25.255445 24878 webserver.cc:460] Webserver started at http://127.16.62.126:40113/ using document root <none> and password file <none>
I20260926 23:17:25.256253 24878 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:25.256400 24878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:25.256783 24878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:25.259681 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "3b566b18797d4acaa60314dea0e0b138"
format_stamp: "Formatted at 2026-09-26 23:17:25 on dist-test-slave-vcv1"
server_key: "def7cbbdb0e307b3acaaccb6673b9a70"
server_key_iv: "f73e000a824edec196fe0bfe2206f78e"
server_key_version: "encryptionkey@0"
I20260926 23:17:25.260509 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "3b566b18797d4acaa60314dea0e0b138"
format_stamp: "Formatted at 2026-09-26 23:17:25 on dist-test-slave-vcv1"
server_key: "def7cbbdb0e307b3acaaccb6673b9a70"
server_key_iv: "f73e000a824edec196fe0bfe2206f78e"
server_key_version: "encryptionkey@0"
I20260926 23:17:25.265552 24878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:17:25.269107 24893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:25.270344 24878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:17:25.270514 24878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "3b566b18797d4acaa60314dea0e0b138"
format_stamp: "Formatted at 2026-09-26 23:17:25 on dist-test-slave-vcv1"
server_key: "def7cbbdb0e307b3acaaccb6673b9a70"
server_key_iv: "f73e000a824edec196fe0bfe2206f78e"
server_key_version: "encryptionkey@0"
I20260926 23:17:25.270699 24878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:25.292645 24878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:25.297004 24878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:25.297333 24878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:25.315717 24878 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:35669
I20260926 23:17:25.315747 24944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:35669 every 8 connection(s)
I20260926 23:17:25.317195 24878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:17:25.317812 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 24878
I20260926 23:17:25.318079 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:17:25.318938 16633 external_mini_cluster.cc:1428] Setting key f4dde1979ac92d998680e69c4d11b05a
I20260926 23:17:25.321511 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:25.331899 24945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: Bootstrap starting.
Sep 26 23:17:25 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:25.335325 24945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:25.336460 24945 log.cc:824] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:25.339769 24945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: No bootstrap required, opened a new log
I20260926 23:17:25.341249 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:25.320961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.16.62.126:35669)
0926 23:17:25.321757 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:25.321769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:25.321812 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:17:25.323433 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:25.323453 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:25.323837 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:25.324264 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:25.325783 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:25.327081 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:25.328352 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:25.329043 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:25.329251 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:25.333434 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:25.333442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:25.333457 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:25.333492 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:25.336666 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:25.337561 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:25.337567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:25.337573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:25.337785 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:25.338156 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:25.338161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:25.338164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:25.338826 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:25.339141 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:25.339463 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 23:17:25.339894 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":336,"threads_started":1}
I20260926 23:17:25.348258 24945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } }
I20260926 23:17:25.348578 24945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:25.348649 24945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b566b18797d4acaa60314dea0e0b138, State: Initialized, Role: FOLLOWER
I20260926 23:17:25.349164 24945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } }
I20260926 23:17:25.349351 24945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:17:25.349445 24945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:17:25.349558 24945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:25.351598 24945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } }
I20260926 23:17:25.352181 24945 leader_election.cc:302] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b566b18797d4acaa60314dea0e0b138; no voters:
I20260926 23:17:25.352690 24945 leader_election.cc:288] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:17:25.352828 24950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:25.353536 24950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 1 LEADER]: Becoming Leader. State: Replica: 3b566b18797d4acaa60314dea0e0b138, State: Running, Role: LEADER
I20260926 23:17:25.354051 24950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } }
I20260926 23:17:25.354714 24945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:25.356572 24952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b566b18797d4acaa60314dea0e0b138. Latest consensus state: current_term: 1 leader_uuid: "3b566b18797d4acaa60314dea0e0b138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } } }
I20260926 23:17:25.356946 24952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:25.356911 24951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b566b18797d4acaa60314dea0e0b138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } } }
I20260926 23:17:25.357290 24951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:25.358850 24955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:17:25.364446 24955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:17:25.365091 24967 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:17:25.372689 24955 catalog_manager.cc:1347] Generated new cluster ID: c41c0049800c4a1ca3f4c3dbd9d405b9
I20260926 23:17:25.372802 24955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:17:25.385799 24955 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:17:25.387733 24955 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:17:25.396731 24955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: Generated new TSK 0
I20260926 23:17:25.397452 24955 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:17:25.456112 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35669
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:25.797670 24971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:25.797948 24971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:25.798054 24971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:25.804957 24971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:25.805051 24971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:25.805160 24971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:17:25.814289 24971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:25.815412 24971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:25.816658 24971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:25.829169 24976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:25.829082 24979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:25.829113 24977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:25.830329 24971 server_base.cc:1034] running on GCE node
I20260926 23:17:25.831180 24971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:25.832199 24971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:25.833443 24971 hybrid_clock.cc:648] HybridClock initialized: now 1790464645833355 us; error 78 us; skew 500 ppm
I20260926 23:17:25.833777 24971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:25 dist-test-slave-vcv1 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464645, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:25.836902 24971 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:17:25.838445 24971 webserver.cc:460] Webserver started at http://127.16.62.65:36401/ using document root <none> and password file <none>
I20260926 23:17:25.839288 24971 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:25.839423 24971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:25.839731 24971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:25.842952 24971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "58cc66b7d6f2495bbbc3469d61cdd49b"
format_stamp: "Formatted at 2026-09-26 23:17:25 on dist-test-slave-vcv1"
server_key: "44b2a24ee3a0b21c6cca0b94806d0d5d"
server_key_iv: "2ddd0aa22b1427a58eef6dcf48027ec2"
server_key_version: "encryptionkey@0"
I20260926 23:17:25.843735 24971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "58cc66b7d6f2495bbbc3469d61cdd49b"
format_stamp: "Formatted at 2026-09-26 23:17:25 on dist-test-slave-vcv1"
server_key: "44b2a24ee3a0b21c6cca0b94806d0d5d"
server_key_iv: "2ddd0aa22b1427a58eef6dcf48027ec2"
server_key_version: "encryptionkey@0"
I20260926 23:17:25.848758 24971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 23:17:25.851939 24986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:25.853404 24971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:25.853574 24971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "58cc66b7d6f2495bbbc3469d61cdd49b"
format_stamp: "Formatted at 2026-09-26 23:17:25 on dist-test-slave-vcv1"
server_key: "44b2a24ee3a0b21c6cca0b94806d0d5d"
server_key_iv: "2ddd0aa22b1427a58eef6dcf48027ec2"
server_key_version: "encryptionkey@0"
I20260926 23:17:25.853794 24971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:25.882122 24971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:25.886446 24971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:25.886732 24971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:25.887670 24971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:25.889236 24971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:25.889334 24971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:25.889490 24971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:25.889566 24971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:25.921633 24971 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:38667
I20260926 23:17:25.921679 25098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:38667 every 8 connection(s)
I20260926 23:17:25.922851 24971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:17:25.929523 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 24971
I20260926 23:17:25.929780 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:17:25.930528 16633 external_mini_cluster.cc:1428] Setting key 6e988864c98a983646e021beaa472777
Sep 26 23:17:25 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464645, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:25.943956 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:25.925343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:41961 (local address 127.16.62.126:35669)
0926 23:17:25.925767 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:25.925772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:25.926722 (+ 950us) server_negotiation.cc:407] Connection header received
0926 23:17:25.928518 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:25.928523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:25.928606 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:25.928716 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:25.930829 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:25.931719 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:25.933891 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:25.934470 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:25.934632 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:25.938553 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:25.938559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:25.938569 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:25.938600 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:25.941089 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:25.941776 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:25.941781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:25.941785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:25.941903 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:25.942303 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:25.942308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:25.942311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:25.942565 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:25.942746 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:25.943358 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 23:17:25.943574 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":213,"threads_started":1}
I20260926 23:17:25.944947 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:25.925904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:41961)
0926 23:17:25.926562 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:25.926603 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:25.928275 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:25.928953 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:25.928960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:25.929789 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:25.930619 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:25.930645 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:25.931879 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:25.931883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:25.933716 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:25.933737 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:25.934792 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:25.934803 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:25.935224 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:25.935996 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:25.936017 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:25.938339 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:25.941227 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:25.941235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:25.941250 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:25.941659 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:25.942005 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:25.942010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:25.942013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:25.942190 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:25.942710 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:25.942720 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:25.943004 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 23:17:25.943253 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 23:17:25.943747 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
I20260926 23:17:25.946498 25099 heartbeater.cc:347] Connected to a master server at 127.16.62.126:35669
I20260926 23:17:25.946883 25099 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:25.947737 25099 heartbeater.cc:511] Master 127.16.62.126:35669 requested a full tablet report, sending...
I20260926 23:17:25.950229 24910 ts_manager.cc:194] Registered new tserver with Master: 58cc66b7d6f2495bbbc3469d61cdd49b (127.16.62.65:38667)
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
I20260926 23:17:25.952394 24910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:41961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:17:25.979274 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35669
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:26.287147 25105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:26.287423 25105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:26.287514 25105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:26.294811 25105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:26.294917 25105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:26.295030 25105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:17:26.304446 25105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:26.305581 25105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:26.306778 25105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:26.320124 25111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:26.320468 25113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:26.320930 25110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:26.321329 25105 server_base.cc:1034] running on GCE node
I20260926 23:17:26.321971 25105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:26.323006 25105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:26.324236 25105 hybrid_clock.cc:648] HybridClock initialized: now 1790464646324178 us; error 53 us; skew 500 ppm
I20260926 23:17:26.324594 25105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:26 dist-test-slave-vcv1 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464646, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:26.328081 25105 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:17:26.334246 25105 webserver.cc:460] Webserver started at http://127.16.62.66:42141/ using document root <none> and password file <none>
I20260926 23:17:26.334905 25105 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:26.335043 25105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:26.335331 25105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:26.338297 25105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "1fc257a4abc84ec6b7a9ba67b905b1f5"
format_stamp: "Formatted at 2026-09-26 23:17:26 on dist-test-slave-vcv1"
server_key: "079add40d2a7afcdba32c7ddb9d5ce3c"
server_key_iv: "bde758e69df34d3d78a1d28b2c92250f"
server_key_version: "encryptionkey@0"
I20260926 23:17:26.339056 25105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "1fc257a4abc84ec6b7a9ba67b905b1f5"
format_stamp: "Formatted at 2026-09-26 23:17:26 on dist-test-slave-vcv1"
server_key: "079add40d2a7afcdba32c7ddb9d5ce3c"
server_key_iv: "bde758e69df34d3d78a1d28b2c92250f"
server_key_version: "encryptionkey@0"
I20260926 23:17:26.344089 25105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 23:17:26.347388 25120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:26.348701 25105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:26.348875 25105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "1fc257a4abc84ec6b7a9ba67b905b1f5"
format_stamp: "Formatted at 2026-09-26 23:17:26 on dist-test-slave-vcv1"
server_key: "079add40d2a7afcdba32c7ddb9d5ce3c"
server_key_iv: "bde758e69df34d3d78a1d28b2c92250f"
server_key_version: "encryptionkey@0"
I20260926 23:17:26.349079 25105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:26.378633 25105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:26.383016 25105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:26.383322 25105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:26.384459 25105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:26.385787 25105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:26.385886 25105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:26.386045 25105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:26.386122 25105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:26.418362 25105 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:34403
I20260926 23:17:26.418396 25232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:34403 every 8 connection(s)
I20260926 23:17:26.419628 25105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:17:26.428659 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 25105
I20260926 23:17:26.428870 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:17:26.429390 16633 external_mini_cluster.cc:1428] Setting key 2db0f76af88d85e79018edf793ffe416
Sep 26 23:17:26 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464646, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:26.439919 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:26.422200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:39859 (local address 127.16.62.126:35669)
0926 23:17:26.422376 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:26.422380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:26.423539 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 23:17:26.425323 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:26.425327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:26.425391 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:26.425478 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:26.426855 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:26.427735 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:26.430042 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:26.430676 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:26.430852 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:26.434219 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:26.434224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:26.434228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:26.434258 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:26.436934 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:26.437663 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:26.437667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:26.437669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:26.437787 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:26.438231 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:26.438235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:26.438237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:26.438474 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:26.438636 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:26.439350 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 23:17:26.439565 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":68}
I20260926 23:17:26.440909 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:26.422713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:39859)
0926 23:17:26.423378 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:26.423418 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:26.425076 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:26.425687 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:26.425694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:26.426173 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:26.426714 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:26.426730 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:26.427890 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:26.427896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:26.429857 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:26.429873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:26.430887 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:26.430892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:26.431084 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:26.431823 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:26.431844 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:26.433990 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:26.437089 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:26.437097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:26.437114 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:26.437517 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:26.437905 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:26.437910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:26.437914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:26.438096 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:26.438600 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:26.438607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:26.438938 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 23:17:26.439256 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 23:17:26.439777 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":285,"threads_started":1}
I20260926 23:17:26.442363 25233 heartbeater.cc:347] Connected to a master server at 127.16.62.126:35669
I20260926 23:17:26.442677 25233 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:26.443506 25233 heartbeater.cc:511] Master 127.16.62.126:35669 requested a full tablet report, sending...
I20260926 23:17:26.445055 24910 ts_manager.cc:194] Registered new tserver with Master: 1fc257a4abc84ec6b7a9ba67b905b1f5 (127.16.62.66:34403)
I20260926 23:17:26.446305 24910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:39859
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:17:26.478720 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35669
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:26.796952 25238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:26.797246 25238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:26.797426 25238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:26.804811 25238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:26.804937 25238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:26.805137 25238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:17:26.814764 25238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:26.816008 25238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:26.817332 25238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:26.830561 25243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:26.830528 25246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:26.830615 25244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:26.831319 25238 server_base.cc:1034] running on GCE node
I20260926 23:17:26.831887 25238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:26.832777 25238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:26.833984 25238 hybrid_clock.cc:648] HybridClock initialized: now 1790464646833940 us; error 39 us; skew 500 ppm
I20260926 23:17:26.834272 25238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:26 dist-test-slave-vcv1 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464646, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:26.837430 25238 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:17:26.838747 25238 webserver.cc:460] Webserver started at http://127.16.62.67:46251/ using document root <none> and password file <none>
I20260926 23:17:26.839375 25238 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:26.839507 25238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:26.839808 25238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:26.842584 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "be4f432175234120b07c14b94cede3c8"
format_stamp: "Formatted at 2026-09-26 23:17:26 on dist-test-slave-vcv1"
server_key: "ae326a2cfaebb6457a50be7959c383cf"
server_key_iv: "f191ef79f34f79af150e84f71cf6a59b"
server_key_version: "encryptionkey@0"
I20260926 23:17:26.843575 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "be4f432175234120b07c14b94cede3c8"
format_stamp: "Formatted at 2026-09-26 23:17:26 on dist-test-slave-vcv1"
server_key: "ae326a2cfaebb6457a50be7959c383cf"
server_key_iv: "f191ef79f34f79af150e84f71cf6a59b"
server_key_version: "encryptionkey@0"
I20260926 23:17:26.849088 25238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 23:17:26.852589 25253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:26.854239 25238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:17:26.854405 25238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "be4f432175234120b07c14b94cede3c8"
format_stamp: "Formatted at 2026-09-26 23:17:26 on dist-test-slave-vcv1"
server_key: "ae326a2cfaebb6457a50be7959c383cf"
server_key_iv: "f191ef79f34f79af150e84f71cf6a59b"
server_key_version: "encryptionkey@0"
I20260926 23:17:26.854664 25238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:26.871384 25238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:26.875231 25238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:26.875510 25238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:26.876488 25238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:26.877794 25238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:26.877894 25238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:26.878042 25238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:26.878111 25238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:26.910400 25238 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:35397
I20260926 23:17:26.910485 25365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:35397 every 8 connection(s)
I20260926 23:17:26.911649 25238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:17:26.917446 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 25238
I20260926 23:17:26.917654 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:17:26.918066 16633 external_mini_cluster.cc:1428] Setting key 84184006d0c19c6f507a945373e9a9e5
Sep 26 23:17:26 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464646, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:26.932231 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:26.914193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:56065 (local address 127.16.62.126:35669)
0926 23:17:26.914377 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:26.914381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:26.915521 (+ 1140us) server_negotiation.cc:407] Connection header received
0926 23:17:26.917502 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:26.917508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:26.917632 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:26.917752 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:26.919475 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:26.920323 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:26.921967 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:26.922403 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:26.922544 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:26.926303 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:26.926309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:26.926312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:26.926344 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:26.929083 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:26.929817 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:26.929825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:26.929831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:26.929971 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:26.930412 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:26.930419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:26.930423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:26.930705 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:26.930885 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:26.931576 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 23:17:26.931796 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:17:26.933238 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:26.914680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:56065)
0926 23:17:26.915363 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:26.915400 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:26.917217 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:26.917985 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:26.917993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:26.918557 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:26.919318 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:26.919338 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:26.920474 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:26.920480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:26.921816 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:26.921829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:26.922644 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:26.922652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:26.922897 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:26.923593 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:26.923622 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:26.926087 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:26.929230 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:26.929239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:26.929255 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:26.929692 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:26.930093 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:26.930100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:26.930104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:26.930297 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:26.930845 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:26.930854 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:26.931196 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 23:17:26.931474 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 23:17:26.931967 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":270,"threads_started":1}
I20260926 23:17:26.934481 25366 heartbeater.cc:347] Connected to a master server at 127.16.62.126:35669
I20260926 23:17:26.934731 25366 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:26.935446 25366 heartbeater.cc:511] Master 127.16.62.126:35669 requested a full tablet report, sending...
I20260926 23:17:26.936714 24910 ts_manager.cc:194] Registered new tserver with Master: be4f432175234120b07c14b94cede3c8 (127.16.62.67:35397)
I20260926 23:17:26.937480 24910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:56065
I20260926 23:17:26.944952 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:17:26.955484 25099 heartbeater.cc:503] Master 127.16.62.126:35669 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab.
Entry for principal oryx/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab.
WARNING: no policy specified for oryx/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab.
Entry for principal oryx/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab.
WARNING: no policy specified for oryx/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab.
Entry for principal oryx/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab.
I20260926 23:17:27.066349 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 24971
Sep 26 23:17:27 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:17:27.082183 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:27.067327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.16.62.65:38667)
0926 23:17:27.067933 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:27.067945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:27.068046 (+ 101us) server_negotiation.cc:407] Connection header received
0926 23:17:27.068299 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:27.068306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:27.068521 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:27.068696 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:27.070038 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:27.071051 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:27.072292 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:27.072858 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:27.072988 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:27.076982 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:27.076992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:27.077005 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:27.077035 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:27.079556 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:27.080234 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:27.080241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:27.080243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:27.080357 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:27.080706 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:27.080710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:27.080712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:27.081012 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:27.081325 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:27.081560 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 23:17:27.081829 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":276,"threads_started":1}
I20260926 23:17:27.083415 25094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:27.164598 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 24971
I20260926 23:17:27.180907 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25105
Sep 26 23:17:27 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:17:27.196310 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:27.181714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.16.62.66:34403)
0926 23:17:27.182279 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:27.182287 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:27.182312 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:17:27.182372 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:27.182378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:27.182592 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:27.182722 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:27.183906 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:27.184931 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:27.186201 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:27.186711 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:27.186811 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:27.190742 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:27.190753 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:27.190766 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:27.190796 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:27.193599 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:27.194234 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:27.194241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:27.194244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:27.194356 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:27.194706 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:27.194712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:27.194716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:27.195027 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:27.195326 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:27.195611 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 23:17:27.195880 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":297,"threads_started":1}
I20260926 23:17:27.197520 25228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:27.277951 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 25105
I20260926 23:17:27.293272 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25238
Sep 26 23:17:27 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:17:27.309340 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:27.294308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.16.62.67:35397)
0926 23:17:27.294881 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:27.294887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:27.294915 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:17:27.294977 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:27.294983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:27.295197 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:27.295368 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:27.296759 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:27.297779 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:27.299090 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:27.299663 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:27.299800 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:27.304181 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:27.304199 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:27.304213 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:27.304246 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:27.306871 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:27.307543 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:27.307550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:27.307552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:27.307672 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:27.307995 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:27.307998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:27.308000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:27.308380 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:27.308607 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:27.308870 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 23:17:27.309055 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":300,"threads_started":1}
I20260926 23:17:27.310556 25361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:27.391175 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 25238
I20260926 23:17:27.406836 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 24878
I20260926 23:17:27.408104 24940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:27.507138 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 24878
I20260926 23:17:27.527357 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:35669
--webserver_interface=127.16.62.126
--webserver_port=40113
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:35669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:27.840655 25383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:27.840967 25383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:27.841120 25383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:27.849164 25383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:27.849280 25383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:27.849355 25383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:27.849433 25383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:27.849496 25383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:27.858762 25383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:35669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:35669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=40113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:27.860015 25383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:27.861299 25383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:27.872627 25388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:27.872680 25391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:27.872582 25389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:27.873433 25383 server_base.cc:1034] running on GCE node
I20260926 23:17:27.874130 25383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:27.875440 25383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:27.876653 25383 hybrid_clock.cc:648] HybridClock initialized: now 1790464647876608 us; error 33 us; skew 500 ppm
I20260926 23:17:27.876981 25383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:27 dist-test-slave-vcv1 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464647, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:27.879882 25383 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:17:27.881314 25383 webserver.cc:460] Webserver started at http://127.16.62.126:40113/ using document root <none> and password file <none>
I20260926 23:17:27.881951 25383 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:27.882062 25383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:27.887171 25383 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 23:17:27.889854 25398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:27.891148 25383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:17:27.891286 25383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "3b566b18797d4acaa60314dea0e0b138"
format_stamp: "Formatted at 2026-09-26 23:17:25 on dist-test-slave-vcv1"
server_key: "def7cbbdb0e307b3acaaccb6673b9a70"
server_key_iv: "f73e000a824edec196fe0bfe2206f78e"
server_key_version: "encryptionkey@0"
I20260926 23:17:27.892262 25383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:27.909816 25383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:27.913638 25383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:27.913856 25383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:27.931470 25383 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:35669
I20260926 23:17:27.931494 25449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:35669 every 8 connection(s)
I20260926 23:17:27.932924 25383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:17:27.936717 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 25383
I20260926 23:17:27.937628 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:38667
--local_ip_for_outbound_sockets=127.16.62.65
--tserver_master_addrs=127.16.62.126:35669
--webserver_port=36401
--webserver_interface=127.16.62.65
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.62.65
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:17:27.940830 25450 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:17:27.945708 25450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: Bootstrap starting.
I20260926 23:17:27.964794 25450 log.cc:824] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:27.973304 25450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:17:27.973820 25450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: Bootstrap complete.
I20260926 23:17:27.982679 25450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } }
I20260926 23:17:27.983436 25450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b566b18797d4acaa60314dea0e0b138, State: Initialized, Role: FOLLOWER
I20260926 23:17:27.984027 25450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } }
I20260926 23:17:27.984302 25450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:17:27.984424 25450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:17:27.984594 25450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:17:27.988046 25450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } }
I20260926 23:17:27.988549 25450 leader_election.cc:302] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b566b18797d4acaa60314dea0e0b138; no voters:
I20260926 23:17:27.989182 25450 leader_election.cc:288] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:17:27.989319 25454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:17:27.990411 25454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [term 2 LEADER]: Becoming Leader. State: Replica: 3b566b18797d4acaa60314dea0e0b138, State: Running, Role: LEADER
I20260926 23:17:27.991016 25454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } }
I20260926 23:17:27.991328 25450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:27.993930 25455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3b566b18797d4acaa60314dea0e0b138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } } }
I20260926 23:17:27.993844 25456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b566b18797d4acaa60314dea0e0b138. Latest consensus state: current_term: 2 leader_uuid: "3b566b18797d4acaa60314dea0e0b138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b566b18797d4acaa60314dea0e0b138" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35669 } } }
I20260926 23:17:27.994360 25455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:27.994371 25456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138 [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:27.998340 25459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:17:28.004881 25459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:17:28.006628 25459 catalog_manager.cc:1259] Loaded cluster ID: c41c0049800c4a1ca3f4c3dbd9d405b9
I20260926 23:17:28.006784 25459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:17:28.011957 25459 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:17:28.013844 25459 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b566b18797d4acaa60314dea0e0b138: Loaded TSK: 0
I20260926 23:17:28.015683 25459 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:17:28.282238 25452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:28.282541 25452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:28.282699 25452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:28.289930 25452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:28.290076 25452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:28.290246 25452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:17:28.300282 25452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.65.keytab
--principal=oryx/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:38667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=36401
--tserver_master_addrs=127.16.62.126:35669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:28.301568 25452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:28.302848 25452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:28.315857 25477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:28.316085 25476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:28.316391 25479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:28.316841 25452 server_base.cc:1034] running on GCE node
I20260926 23:17:28.317569 25452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:28.318791 25452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:28.320041 25452 hybrid_clock.cc:648] HybridClock initialized: now 1790464648319982 us; error 60 us; skew 500 ppm
I20260926 23:17:28.320449 25452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:28 dist-test-slave-vcv1 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464648, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:28.324354 25452 init.cc:377] Logged in from keytab as oryx/127.16.62.65@KRBTEST.COM (short username oryx)
I20260926 23:17:28.325860 25452 webserver.cc:460] Webserver started at http://127.16.62.65:36401/ using document root <none> and password file <none>
I20260926 23:17:28.326531 25452 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:28.326682 25452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:28.332551 25452 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:17:28.337773 25486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:28.340463 25452 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260926 23:17:28.340698 25452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "58cc66b7d6f2495bbbc3469d61cdd49b"
format_stamp: "Formatted at 2026-09-26 23:17:25 on dist-test-slave-vcv1"
server_key: "44b2a24ee3a0b21c6cca0b94806d0d5d"
server_key_iv: "2ddd0aa22b1427a58eef6dcf48027ec2"
server_key_version: "encryptionkey@0"
I20260926 23:17:28.341953 25452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:28.380736 25452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:28.386842 25452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:28.387184 25452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:28.388543 25452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:28.390476 25452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:28.390606 25452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:28.390746 25452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:28.390852 25452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:28.424085 25452 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:38667
I20260926 23:17:28.424096 25598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:38667 every 8 connection(s)
I20260926 23:17:28.426206 25452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:17:28.432427 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 25452
I20260926 23:17:28.433008 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:34403
--local_ip_for_outbound_sockets=127.16.62.66
--tserver_master_addrs=127.16.62.126:35669
--webserver_port=42141
--webserver_interface=127.16.62.66
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.62.66
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:17:28 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:28 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:17:28.912612 25603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:28.912883 25603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:28.913024 25603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:28.920369 25603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:28.920486 25603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:28.920663 25603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:17:28.930035 25603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.66.keytab
--principal=oryx/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:34403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=42141
--tserver_master_addrs=127.16.62.126:35669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:28.931412 25603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:28.932801 25603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:28.949344 25609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:28.954604 25610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:28.955421 25612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:28.956452 25603 server_base.cc:1034] running on GCE node
I20260926 23:17:28.957088 25603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:28.958174 25603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:28.959555 25603 hybrid_clock.cc:648] HybridClock initialized: now 1790464648959459 us; error 86 us; skew 500 ppm
I20260926 23:17:28.959858 25603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:28 dist-test-slave-vcv1 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464648, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:28.963037 25603 init.cc:377] Logged in from keytab as oryx/127.16.62.66@KRBTEST.COM (short username oryx)
I20260926 23:17:28.964684 25603 webserver.cc:460] Webserver started at http://127.16.62.66:42141/ using document root <none> and password file <none>
I20260926 23:17:28.965346 25603 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:28.965484 25603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:28.970796 25603 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:17:28.973431 25619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:28.974742 25603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:28.974902 25603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "1fc257a4abc84ec6b7a9ba67b905b1f5"
format_stamp: "Formatted at 2026-09-26 23:17:26 on dist-test-slave-vcv1"
server_key: "079add40d2a7afcdba32c7ddb9d5ce3c"
server_key_iv: "bde758e69df34d3d78a1d28b2c92250f"
server_key_version: "encryptionkey@0"
I20260926 23:17:28.975795 25603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:29.003782 25603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:29.008059 25603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:29.008368 25603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:29.009418 25603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:29.010809 25603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:29.010893 25603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:29.011020 25603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:29.011086 25603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:29.054459 25603 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:34403
I20260926 23:17:29.054509 25731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:34403 every 8 connection(s)
I20260926 23:17:29.055917 25603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:17:29.059685 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 25603
I20260926 23:17:29.060433 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:35397
--local_ip_for_outbound_sockets=127.16.62.67
--tserver_master_addrs=127.16.62.126:35669
--webserver_port=46251
--webserver_interface=127.16.62.67
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.62.67
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:17:29 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:29 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:17:29.647711 25736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:29.648165 25736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:29.648378 25736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:29.660053 25736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:29.660207 25736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:29.660456 25736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
W20260926 23:17:29.672441 25601 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.229s user 0.006s sys 0.014s
I20260926 23:17:29.674890 25736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/krb5kdc/oryx_127.16.62.67.keytab
--principal=oryx/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:35397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=46251
--tserver_master_addrs=127.16.62.126:35669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
W20260926 23:17:29.675225 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:28.429330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:37565 (local address 127.16.62.126:35669)
0926 23:17:28.430840 (+ 1510us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:28.430853 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:28.430900 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:17:28.432788 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:28.432811 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:28.433299 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:28.433722 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:28.435603 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:28.438023 (+ 2420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:28.439924 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:28.440637 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:28.441984 (+ 1347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:29.674165 (+1232181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:37565: BlockingRecv error: recv got EOF from 127.16.62.65:37565 (error 108)
Metrics: {"server-negotiator.queue_time_us":1267,"thread_start_us":1042,"threads_started":1}
I20260926 23:17:29.676220 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:28.429297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:37565)
0926 23:17:28.430066 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:28.430113 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:28.432056 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:28.433884 (+ 1828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:28.433897 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:28.434645 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:28.435396 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:28.435421 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:28.438203 (+ 2782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:28.438211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:28.439790 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:28.439802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:28.442018 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:28.442026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:28.442286 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:28.442908 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:28.442933 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:29.673836 (+1230903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":298,"threads_started":1}
I20260926 23:17:29.676614 25736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260926 23:17:29.677232 25599 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:35669 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.62.126:35669: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
I20260926 23:17:29.678211 25736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 26 23:17:29 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:29 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:29.699246 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:29.678081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:44845)
0926 23:17:29.680558 (+ 2477us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:29.680589 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:29.680794 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:29.681276 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:29.681285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:29.682001 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:29.682323 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:29.682342 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.683818 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.683828 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:29.686409 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:29.686435 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.688204 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.688216 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:29.688533 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:29.689395 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:29.689435 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:29.698710 (+ 9275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2254,"thread_start_us":2831,"threads_started":1}
W20260926 23:17:29.700237 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:29.678628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:44845 (local address 127.16.62.126:35669)
0926 23:17:29.678871 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:29.678888 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:29.680685 (+ 1797us) server_negotiation.cc:407] Connection header received
0926 23:17:29.680969 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:29.680974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:29.681053 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:29.681161 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:29.682518 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.683635 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.686709 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.687890 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.688176 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:29.699870 (+ 11694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:44845: BlockingRecv error: recv got EOF from 127.16.62.65:44845 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260926 23:17:29.702965 25744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:29.703646 25743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:29.705895 25746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:29.706734 25736 server_base.cc:1034] running on GCE node
I20260926 23:17:29.707412 25736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:29.708626 25736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:29.709993 25736 hybrid_clock.cc:648] HybridClock initialized: now 1790464649709917 us; error 71 us; skew 500 ppm
I20260926 23:17:29.710374 25736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:29 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:29 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:29 dist-test-slave-vcv1 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464649, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:29.714607 25736 init.cc:377] Logged in from keytab as oryx/127.16.62.67@KRBTEST.COM (short username oryx)
W20260926 23:17:29.716128 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:29.702647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:55249 (local address 127.16.62.126:35669)
0926 23:17:29.703552 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:29.703562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:29.703590 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:17:29.703666 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:29.703675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:29.703785 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:29.703934 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:29.705874 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.707536 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.709291 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.709652 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.709794 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:29.715900 (+ 6106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:55249: BlockingRecv error: recv got EOF from 127.16.62.65:55249 (error 108)
Metrics: {"server-negotiator.queue_time_us":720}
I20260926 23:17:29.716241 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:29.702108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:55249)
0926 23:17:29.703027 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:29.703056 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:29.703227 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:29.704202 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:29.704211 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:29.705108 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:29.705639 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:29.705654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.707726 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.707733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:29.709155 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:29.709166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.709834 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.709840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:29.710025 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:29.710831 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:29.710856 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:29.715705 (+ 4849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":711}
W20260926 23:17:29.716817 25599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:17:29.720475 25736 webserver.cc:460] Webserver started at http://127.16.62.67:46251/ using document root <none> and password file <none>
I20260926 23:17:29.721149 25736 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:29.721293 25736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:29.727223 25736 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 23:17:29.731863 25753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:29.734045 25736 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 23:17:29.734293 25736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "be4f432175234120b07c14b94cede3c8"
format_stamp: "Formatted at 2026-09-26 23:17:26 on dist-test-slave-vcv1"
server_key: "ae326a2cfaebb6457a50be7959c383cf"
server_key_iv: "f191ef79f34f79af150e84f71cf6a59b"
server_key_version: "encryptionkey@0"
I20260926 23:17:29.736450 25736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:29.801913 25736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:29.809710 25736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:29.810110 25736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:29.811650 25736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:29.814118 25736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:29.814278 25736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:29.814426 25736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:29.814553 25736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:29.852233 25736 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:35397
I20260926 23:17:29.852242 25865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:35397 every 8 connection(s)
I20260926 23:17:29.854773 25736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:17:29.858112 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 25736
Sep 26 23:17:29 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:29 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:29.889151 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:29.859149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.16.62.126:35669)
0926 23:17:29.867919 (+ 8770us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:29.867925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:29.868581 (+ 656us) server_negotiation.cc:407] Connection header received
0926 23:17:29.868745 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:29.868771 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:29.868890 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:29.869046 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:29.871282 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.872425 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.873957 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.874908 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.875072 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:29.879627 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:29.879640 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:29.879664 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:29.879720 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:29.884213 (+ 4493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:29.885278 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:29.885289 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:29.885300 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:29.885501 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:29.886488 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:29.886494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:29.886497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:29.887182 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:29.887498 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:29.888184 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 23:17:29.888594 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":8630,"thread_start_us":9859,"threads_started":1}
W20260926 23:17:30.272109 25735 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.197s user 0.010s sys 0.015s
I20260926 23:17:30.273933 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:29.059298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:51391)
0926 23:17:29.060423 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:29.060493 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:29.064316 (+ 3823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:29.065400 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:29.065418 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:29.066465 (+ 1047us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:29.067602 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:29.067636 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.069374 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.069382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:29.072468 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:29.072491 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.073733 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.073743 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:29.074167 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:29.075405 (+ 1238us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:29.075447 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:30.272983 (+1197536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":403,"threads_started":1}
W20260926 23:17:30.273998 25734 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:29.058718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:51391 (local address 127.16.62.126:35669)
0926 23:17:29.059324 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:29.059335 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:29.060677 (+ 1342us) server_negotiation.cc:407] Connection header received
0926 23:17:29.064758 (+ 4081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:29.064766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:29.064859 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:29.064985 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:29.067848 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.069204 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.072675 (+ 3471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.073429 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.073581 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.273446 (+1199865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:51391: BlockingRecv error: recv got EOF from 127.16.62.66:51391 (error 108)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":272,"threads_started":1}
W20260926 23:17:30.275281 25732 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:35669 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.62.126:35669: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:30.294039 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:30.276109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:50259)
0926 23:17:30.276374 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:30.276402 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:30.276578 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:30.276930 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:30.276938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:30.277591 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:30.277904 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.277921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.278983 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.278992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.281372 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.281395 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.282174 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.282183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.282455 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.288454 (+ 5999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:30.288492 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:30.293734 (+ 5242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 23:17:30.294148 25734 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:30.276222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:50259 (local address 127.16.62.126:35669)
0926 23:17:30.276377 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:30.276382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:30.276486 (+ 104us) server_negotiation.cc:407] Connection header received
0926 23:17:30.276718 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:30.276721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:30.276779 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:30.276856 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:30.278075 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.278833 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.281539 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.281971 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.282064 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.293921 (+ 11857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:50259: BlockingRecv error: recv got EOF from 127.16.62.66:50259 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:30.305377 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:30.295049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:41565)
0926 23:17:30.295326 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:30.295344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:30.295469 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:30.296023 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:30.296026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:30.296398 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:30.296606 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.296615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.298547 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.298550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.299457 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.299468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.300806 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.300810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.300945 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.301497 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:30.301513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:30.305097 (+ 3584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178}
W20260926 23:17:30.305552 25734 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:30.295329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:41565 (local address 127.16.62.126:35669)
0926 23:17:30.295594 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:30.295604 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:30.295640 (+ 36us) server_negotiation.cc:407] Connection header received
0926 23:17:30.295715 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:30.295723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:30.295856 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:30.296003 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:30.296745 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.298393 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.299595 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.300602 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.300810 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.305266 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:41565: BlockingRecv error: recv got EOF from 127.16.62.66:41565 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 23:17:30.305929 25732 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:30.728804 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:30.717591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:59651)
0926 23:17:30.718214 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:30.718230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:30.718384 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:30.718751 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:30.718755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:30.719360 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:30.719546 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.719558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.721714 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.721720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.722802 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.722816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.724286 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.724291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.724421 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.725118 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:30.725137 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:30.728532 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":362,"threads_started":1}
W20260926 23:17:30.729558 25734 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:30.717586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:59651 (local address 127.16.62.126:35669)
0926 23:17:30.717747 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:30.717751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:30.718310 (+ 559us) server_negotiation.cc:407] Connection header received
0926 23:17:30.718504 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:30.718509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:30.718595 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:30.718712 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:30.719736 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.721472 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.722981 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.724027 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.724432 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.729204 (+ 4772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:59651: BlockingRecv error: recv got EOF from 127.16.62.65:59651 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 23:17:30.974570 25868 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.098s user 0.001s sys 0.014s
W20260926 23:17:30.975663 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:29.857848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:49863 (local address 127.16.62.126:35669)
0926 23:17:29.858137 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:29.858146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:29.865621 (+ 7475us) server_negotiation.cc:407] Connection header received
0926 23:17:29.868974 (+ 3353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:29.868979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:29.869054 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:29.869160 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:29.871258 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.873025 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.874855 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.875594 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.875728 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.975422 (+1099694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:49863: BlockingRecv error: recv got EOF from 127.16.62.67:49863 (error 108)
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":101}
I20260926 23:17:30.976372 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:29.858361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:49863)
0926 23:17:29.865454 (+ 7093us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:29.865500 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:29.868684 (+ 3184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:29.869397 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:29.869408 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:29.870162 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:29.871051 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:29.871076 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.873189 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.873218 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:29.874711 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:29.874723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:29.875827 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:29.875834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:29.876052 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:29.876709 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:29.876730 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:30.975281 (+1098551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6800,"thread_start_us":6588,"threads_started":1}
W20260926 23:17:30.977288 25866 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:35669 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.62.126:35669: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:30.986809 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:30.977974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:59789)
0926 23:17:30.978216 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:30.978232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:30.978326 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:30.978618 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:30.978621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:30.978994 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:30.979237 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.979249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.980317 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.980322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.981260 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.981270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.981998 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.982003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.982188 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.982840 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:30.982857 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:30.986525 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 23:17:30.986979 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:30.978059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:59789 (local address 127.16.62.126:35669)
0926 23:17:30.978257 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:30.978261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:30.978276 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:17:30.978424 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:30.978427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:30.978483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:30.978565 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:30.979377 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.980199 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.981386 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.981820 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.981951 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.986718 (+ 4767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:59789: BlockingRecv error: recv got EOF from 127.16.62.67:59789 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:30 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:30.996902 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:30.987566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:33231)
0926 23:17:30.987755 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:30.987777 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:30.987889 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:30.988270 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:30.988276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:30.988652 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:30.988811 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.988821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.989928 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.989935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.991056 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:30.991068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.991901 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.991907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:30.992215 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.992746 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:30.992765 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:30.996562 (+ 3797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 23:17:30.997028 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:30.987652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:33231 (local address 127.16.62.126:35669)
0926 23:17:30.987842 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:30.987850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:30.987870 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:17:30.988030 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:30.988039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:30.988115 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:30.988249 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:30.988961 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.989783 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.991213 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:30.991714 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:30.991906 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:30.996776 (+ 4870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:33231: BlockingRecv error: recv got EOF from 127.16.62.67:33231 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 23:17:30.997236 25866 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 23:17:31 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:31 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:31.315518 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:31.306499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:49961)
0926 23:17:31.307147 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:31.307164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:31.307286 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:31.307573 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:31.307576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:31.307993 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:31.308231 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:31.308240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:31.309189 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:31.309193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:31.310273 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:31.310285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:31.310995 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:31.311000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:31.311155 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:31.311711 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:31.311728 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:31.315277 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":372,"threads_started":1}
W20260926 23:17:31.315776 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:31.306556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:49961 (local address 127.16.62.126:35669)
0926 23:17:31.306720 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:31.306724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:31.307236 (+ 512us) server_negotiation.cc:407] Connection header received
0926 23:17:31.307383 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:31.307386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:31.307445 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:31.307524 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:31.308356 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:31.309047 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:31.310405 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:31.310808 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:31.310929 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:31.315492 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:49961: BlockingRecv error: recv got EOF from 127.16.62.66:49961 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 23:17:31 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:31 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:31.738679 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:31.729635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:34287)
0926 23:17:31.730105 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:31.730120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:31.730237 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:31.730558 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:31.730562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:31.730943 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:31.731093 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:31.731101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:31.732028 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:31.732031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:31.732958 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:31.732968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:31.733993 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:31.733999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:31.734168 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:31.734873 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:31.734908 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:31.738450 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":269,"threads_started":1}
W20260926 23:17:31.738865 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:31.729751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:34287 (local address 127.16.62.126:35669)
0926 23:17:31.729950 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:31.729956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:31.730185 (+ 229us) server_negotiation.cc:407] Connection header received
0926 23:17:31.730363 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:31.730367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:31.730429 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:31.730532 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:31.731217 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:31.731920 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:31.733097 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:31.733818 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:31.733968 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:31.738641 (+ 4673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:34287: BlockingRecv error: recv got EOF from 127.16.62.65:34287 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 23:17:31 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:32 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:32.006201 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:31.997792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:46709)
0926 23:17:31.998263 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:31.998278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:31.998388 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:31.998652 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:31.998655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:31.999053 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:31.999189 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:31.999197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.000204 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.000208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:32.001236 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:32.001247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.001857 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.001861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:32.001969 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:32.002473 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:32.002489 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:32.005882 (+ 3393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":268,"threads_started":1}
W20260926 23:17:32.006299 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:31.997828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:46709 (local address 127.16.62.126:35669)
0926 23:17:31.998043 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:31.998052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:31.998354 (+ 302us) server_negotiation.cc:407] Connection header received
0926 23:17:31.998479 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:31.998482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:31.998540 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:31.998617 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:31.999300 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.000037 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.001349 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.001708 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.001809 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:32.006056 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:46709: BlockingRecv error: recv got EOF from 127.16.62.67:46709 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 23:17:32 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:32 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:32.325490 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:32.316507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:54991)
0926 23:17:32.316998 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:32.317014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:32.317124 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:32.317426 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:32.317429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:32.317862 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:32.318036 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:32.318044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.319131 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.319136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:32.320080 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:32.320090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.321004 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.321011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:32.321200 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:32.321833 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:32.321859 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:32.325144 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":289,"threads_started":1}
W20260926 23:17:32.325562 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:32.316542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:54991 (local address 127.16.62.126:35669)
0926 23:17:32.316770 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:32.316780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:32.317094 (+ 314us) server_negotiation.cc:407] Connection header received
0926 23:17:32.317246 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:32.317251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:32.317311 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:32.317394 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:32.318167 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.319010 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.320319 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.320849 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.320987 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:32.325347 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:54991: BlockingRecv error: recv got EOF from 127.16.62.66:54991 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 23:17:32 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:32 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:32.748281 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:32.739694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:41013)
0926 23:17:32.740220 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:32.740245 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:32.740355 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:32.740636 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:32.740639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:32.740969 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:32.741121 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:32.741128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.742204 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.742210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:32.743127 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:32.743141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.743874 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.743880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:32.744053 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:32.744685 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:32.744703 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:32.747924 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
W20260926 23:17:32.748420 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:32.739799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:41013 (local address 127.16.62.126:35669)
0926 23:17:32.740064 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:32.740070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:32.740313 (+ 243us) server_negotiation.cc:407] Connection header received
0926 23:17:32.740456 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:32.740460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:32.740527 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:32.740605 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:32.741245 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.742060 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.743297 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:32.743737 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:32.743860 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:32.748158 (+ 4298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:41013: BlockingRecv error: recv got EOF from 127.16.62.65:41013 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 23:17:33 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:33 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:33.015928 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:33.007212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:34505)
0926 23:17:33.007687 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:33.007703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:33.007852 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:33.008226 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:33.008231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:33.008652 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:33.008840 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:33.008848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.009937 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.009944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:33.010893 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:33.010904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.011580 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.011586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:33.011768 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:33.012311 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:33.012328 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:33.015573 (+ 3245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":271,"threads_started":1}
W20260926 23:17:33.015992 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:33.007289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:34505 (local address 127.16.62.126:35669)
0926 23:17:33.007503 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:33.007511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:33.007786 (+ 275us) server_negotiation.cc:407] Connection header received
0926 23:17:33.007970 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:33.007980 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:33.008051 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:33.008171 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:33.008978 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.009816 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.011026 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.011438 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.011565 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:33.015772 (+ 4207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:34505: BlockingRecv error: recv got EOF from 127.16.62.67:34505 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 23:17:33 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:33 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:33.334971 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:33.326403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:34769)
0926 23:17:33.326873 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:33.326887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:33.327033 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:33.327349 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:33.327354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:33.327714 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:33.327883 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:33.327891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.328951 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.328957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:33.329853 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:33.329864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.330542 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.330548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:33.330734 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:33.331348 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:33.331366 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:33.334747 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":236,"threads_started":1}
W20260926 23:17:33.335166 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:33.326451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:34769 (local address 127.16.62.126:35669)
0926 23:17:33.326649 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:33.326657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:33.326967 (+ 310us) server_negotiation.cc:407] Connection header received
0926 23:17:33.327146 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:33.327152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:33.327221 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:33.327325 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:33.328019 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.328813 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.329983 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.330408 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.330533 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:33.334922 (+ 4389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:34769: BlockingRecv error: recv got EOF from 127.16.62.66:34769 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 23:17:33 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:33 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:33.758071 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:33.749194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:33183)
0926 23:17:33.749677 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:33.749693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:33.749804 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:33.750150 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:33.750153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:33.750525 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:33.750714 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:33.750723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.751741 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.751747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:33.752861 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:33.752874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.753539 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.753543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:33.753689 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:33.754213 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:33.754247 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:33.757758 (+ 3511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":248,"threads_started":1}
W20260926 23:17:33.758174 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:33.749251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:33183 (local address 127.16.62.126:35669)
0926 23:17:33.749460 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:33.749469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:33.749769 (+ 300us) server_negotiation.cc:407] Connection header received
0926 23:17:33.749894 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:33.749897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:33.749956 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:33.750032 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:33.750861 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.751592 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.752989 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:33.753416 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:33.753513 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:33.757981 (+ 4468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:33183: BlockingRecv error: recv got EOF from 127.16.62.65:33183 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 23:17:34 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:34 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:34.025179 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:34.016829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:45415)
0926 23:17:34.017366 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:34.017382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:34.017489 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:34.017893 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:34.017896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:34.018245 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:34.018411 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:34.018418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.019425 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.019429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:34.020350 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:34.020361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.020965 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.020969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:34.021089 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:34.021636 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:34.021654 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:34.024817 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":295,"threads_started":1}
W20260926 23:17:34.025309 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:34.016916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:45415 (local address 127.16.62.126:35669)
0926 23:17:34.017084 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:34.017088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:34.017434 (+ 346us) server_negotiation.cc:407] Connection header received
0926 23:17:34.017641 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:34.017647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:34.017742 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:34.017848 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:34.018524 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.019307 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.020475 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.020851 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.020946 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:34.025028 (+ 4082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:45415: BlockingRecv error: recv got EOF from 127.16.62.67:45415 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 23:17:34 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:34 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:34.344926 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:34.336012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:47521)
0926 23:17:34.336572 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:34.336588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:34.336694 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:34.336984 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:34.336987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:34.337360 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:34.337511 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:34.337519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.338622 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.338625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:34.339800 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:34.339825 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.340713 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.340718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:34.340887 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:34.341387 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:34.341402 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:34.344584 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":328,"threads_started":1}
W20260926 23:17:34.345041 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:34.336066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:47521 (local address 127.16.62.126:35669)
0926 23:17:34.336303 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:34.336311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:34.336686 (+ 375us) server_negotiation.cc:407] Connection header received
0926 23:17:34.336795 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:34.336799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:34.336860 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:34.336936 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:34.337676 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.338483 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.339945 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.340555 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.340703 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:34.344804 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:47521: BlockingRecv error: recv got EOF from 127.16.62.66:47521 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 23:17:34 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:34 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:34.767472 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:34.759014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:36091)
0926 23:17:34.759616 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:34.759636 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:34.759790 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:34.760201 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:34.760206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:34.760678 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:34.760881 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:34.760892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.761910 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.761914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:34.762862 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:34.762872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.763469 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.763473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:34.763588 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:34.764201 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:34.764224 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:34.767253 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":331,"threads_started":1}
W20260926 23:17:34.767637 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:34.759110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:36091 (local address 127.16.62.126:35669)
0926 23:17:34.759285 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:34.759290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:34.759735 (+ 445us) server_negotiation.cc:407] Connection header received
0926 23:17:34.759942 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:34.759948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:34.760019 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:34.760121 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:34.761023 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.761745 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.762977 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:34.763353 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:34.763443 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:34.767425 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:36091: BlockingRecv error: recv got EOF from 127.16.62.65:36091 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 23:17:35 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:35 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:35.034902 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:35.026244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:41515)
0926 23:17:35.026769 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:35.026785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:35.026886 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:35.027190 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:35.027193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:35.027559 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:35.027730 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:35.027738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.028981 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.028991 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:35.029954 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:35.029965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.030596 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.030602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:35.030794 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:35.031393 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:35.031411 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:35.034544 (+ 3133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":300,"threads_started":1}
W20260926 23:17:35.034931 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:35.026390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:41515 (local address 127.16.62.126:35669)
0926 23:17:35.026553 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:35.026557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:35.026850 (+ 293us) server_negotiation.cc:407] Connection header received
0926 23:17:35.026973 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:35.026976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:35.027066 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:35.027148 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:35.027913 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.028843 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.030091 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.030462 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.030586 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:35.034723 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:41515: BlockingRecv error: recv got EOF from 127.16.62.67:41515 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 23:17:35 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:35 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:35.359277 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:35.345873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:59321)
0926 23:17:35.346401 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:35.346415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:35.346521 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:35.346794 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:35.346797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:35.347194 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:35.347385 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:35.347394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.348300 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.348303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:35.349216 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:35.349226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.349882 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.349885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:35.349993 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:35.355824 (+ 5831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:35.355841 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:35.359054 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":297,"threads_started":1}
W20260926 23:17:35.359521 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:35.346054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:59321 (local address 127.16.62.126:35669)
0926 23:17:35.346225 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:35.346229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:35.346487 (+ 258us) server_negotiation.cc:407] Connection header received
0926 23:17:35.346612 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:35.346616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:35.346677 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:35.346758 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:35.347497 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.348193 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.349340 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.349769 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.349862 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:35.359253 (+ 9391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:59321: BlockingRecv error: recv got EOF from 127.16.62.66:59321 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 23:17:35 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:35 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:35.777153 25886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:35.768440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:59313)
0926 23:17:35.768917 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:35.768932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:35.769095 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:35.769369 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:35.769372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:35.769750 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:35.769904 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:35.769912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.770791 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.770794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:35.771729 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:35.771740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.772431 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.772437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:35.772575 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:35.773125 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:35.773144 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:35.776799 (+ 3655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":261,"threads_started":1}
W20260926 23:17:35.777199 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:35.768505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:59313 (local address 127.16.62.126:35669)
0926 23:17:35.768756 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:35.768762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:35.769026 (+ 264us) server_negotiation.cc:407] Connection header received
0926 23:17:35.769183 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:35.769188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:35.769271 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:35.769355 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:35.770016 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.770691 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.771853 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:35.772262 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:35.772355 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:35.776986 (+ 4631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:59313: BlockingRecv error: recv got EOF from 127.16.62.65:59313 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 23:17:36 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:36 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:36.043983 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:36.035776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:43019)
0926 23:17:36.036302 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:36.036318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:36.036423 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:36.036709 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:36.036712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:36.037126 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:36.037263 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:36.037271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.038210 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.038214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:36.039119 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:36.039129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.039761 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.039765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:36.039885 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:36.040424 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:36.040441 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:36.043771 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":240,"threads_started":1}
W20260926 23:17:36.044178 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:36.035812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:43019 (local address 127.16.62.126:35669)
0926 23:17:36.036021 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:36.036031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:36.036391 (+ 360us) server_negotiation.cc:407] Connection header received
0926 23:17:36.036515 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:36.036519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:36.036588 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:36.036676 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:36.037380 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.038101 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.039228 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.039647 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.039735 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:36.043935 (+ 4200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:43019: BlockingRecv error: recv got EOF from 127.16.62.67:43019 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 23:17:36 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:36 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:36.368960 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:36.360116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:34813)
0926 23:17:36.360659 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:36.360675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:36.360824 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:36.361202 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:36.361206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:36.361757 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:36.361988 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:36.361998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.363027 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.363030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:36.363973 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:36.363997 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.364678 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.364682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:36.364811 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:36.365386 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:36.365403 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:36.368623 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":277,"threads_started":1}
W20260926 23:17:36.369038 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:36.360247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:34813 (local address 127.16.62.126:35669)
0926 23:17:36.360420 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:36.360424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:36.360770 (+ 346us) server_negotiation.cc:407] Connection header received
0926 23:17:36.360970 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:36.360976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:36.361065 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:36.361171 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:36.362179 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.362909 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.364109 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.364529 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.364622 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:36.368825 (+ 4203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:34813: BlockingRecv error: recv got EOF from 127.16.62.66:34813 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 23:17:36 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:36 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:36.786940 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:36.778029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:34331)
0926 23:17:36.778616 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:36.778631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:36.778747 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:36.779083 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:36.779086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:36.779451 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:36.779606 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:36.779613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.780671 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.780675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:36.781562 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:36.781611 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.782589 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.782602 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:36.782764 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:36.783375 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:36.783391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:36.786551 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":310,"threads_started":1}
W20260926 23:17:36.786993 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:36.778056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:34331 (local address 127.16.62.126:35669)
0926 23:17:36.778250 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:36.778254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:36.778703 (+ 449us) server_negotiation.cc:407] Connection header received
0926 23:17:36.778876 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:36.778880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:36.778938 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:36.779040 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:36.779720 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.780569 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.781735 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:36.782375 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:36.782475 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:36.786762 (+ 4287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:34331: BlockingRecv error: recv got EOF from 127.16.62.65:34331 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 23:17:37 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:37 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:37.053660 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:37.044861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:59031)
0926 23:17:37.045365 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:37.045381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:37.045491 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:37.045775 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:37.045778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:37.046168 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:37.046325 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:37.046333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.047372 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.047378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:37.048386 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:37.048398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.049119 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.049125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:37.049304 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:37.049905 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:37.049922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:37.053453 (+ 3531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":293,"threads_started":1}
W20260926 23:17:37.053818 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:37.044915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:59031 (local address 127.16.62.126:35669)
0926 23:17:37.045165 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:37.045170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:37.045451 (+ 281us) server_negotiation.cc:407] Connection header received
0926 23:17:37.045584 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:37.045588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:37.045646 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:37.045737 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:37.046464 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.047214 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.048526 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.048958 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.049097 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:37.053628 (+ 4531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:59031: BlockingRecv error: recv got EOF from 127.16.62.67:59031 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 23:17:37 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:37 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:37.378135 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:37.369814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:35227)
0926 23:17:37.370357 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:37.370373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:37.370480 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:37.370757 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:37.370760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:37.371114 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:37.371284 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:37.371292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.372237 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.372240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:37.373163 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:37.373174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.373784 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.373788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:37.373896 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:37.374527 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:37.374544 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:37.377900 (+ 3356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":261,"threads_started":1}
W20260926 23:17:37.378304 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:37.369872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:35227 (local address 127.16.62.126:35669)
0926 23:17:37.370127 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:37.370136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:37.370447 (+ 311us) server_negotiation.cc:407] Connection header received
0926 23:17:37.370570 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:37.370574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:37.370630 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:37.370706 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:37.371407 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.372082 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.373274 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.373668 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.373762 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:37.378082 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:35227: BlockingRecv error: recv got EOF from 127.16.62.66:35227 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 23:17:37 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:37 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:37.797219 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:37.787908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:51607)
0926 23:17:37.788462 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:37.788478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:37.788636 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:37.788962 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:37.788966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:37.789409 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:37.789588 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:37.789598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.790605 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.790611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:37.791670 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:37.791684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.792741 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.792747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:37.792945 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:37.793652 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:37.793668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:37.796982 (+ 3314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":317,"threads_started":1}
W20260926 23:17:37.797463 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:37.788004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:51607 (local address 127.16.62.126:35669)
0926 23:17:37.788224 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:37.788233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:37.788564 (+ 331us) server_negotiation.cc:407] Connection header received
0926 23:17:37.788758 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:37.788765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:37.788830 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:37.788933 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:37.789750 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.790464 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.791838 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:37.792543 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:37.792712 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:37.797206 (+ 4494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:51607: BlockingRecv error: recv got EOF from 127.16.62.65:51607 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 23:17:38 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:38 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:38.063457 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:38.054692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:40925)
0926 23:17:38.055259 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:38.055274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:38.055426 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:38.055787 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:38.055792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:38.056304 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:38.056481 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:38.056489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.057448 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.057455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:38.058389 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:38.058399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.059140 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.059145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:38.059308 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:38.059816 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:38.059833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:38.063148 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":292,"threads_started":1}
W20260926 23:17:38.063556 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:38.054797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:40925 (local address 127.16.62.126:35669)
0926 23:17:38.054984 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:38.054994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:38.055363 (+ 369us) server_negotiation.cc:407] Connection header received
0926 23:17:38.055545 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:38.055551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:38.055622 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:38.055721 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:38.056622 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.057312 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.058527 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.058995 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.059128 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:38.063326 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:40925: BlockingRecv error: recv got EOF from 127.16.62.67:40925 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 23:17:38 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:38 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:38.388490 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:38.379027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:41567)
0926 23:17:38.379609 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:38.379645 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:38.379798 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:38.380277 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:38.380282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:38.380735 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:38.380901 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:38.380909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.382178 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.382185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:38.383226 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:38.383237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.384013 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.384020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:38.384219 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:38.384909 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:38.384927 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:38.388258 (+ 3331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":269,"threads_started":1}
W20260926 23:17:38.388607 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:38.379076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:41567 (local address 127.16.62.126:35669)
0926 23:17:38.379299 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:38.379309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:38.379733 (+ 424us) server_negotiation.cc:407] Connection header received
0926 23:17:38.379944 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:38.379950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:38.380107 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:38.380249 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:38.381047 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.382018 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.383385 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.383829 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.383974 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:38.388397 (+ 4423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:41567: BlockingRecv error: recv got EOF from 127.16.62.66:41567 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 23:17:38 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:38 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:38.806691 25895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:38.798059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:59457)
0926 23:17:38.798599 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:38.798615 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:38.798719 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:38.798995 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:38.798998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:38.799386 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:38.799527 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:38.799534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.800551 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.800555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:38.801574 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:38.801585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.802175 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.802179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:38.802298 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:38.802961 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:38.802978 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:38.806378 (+ 3400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":299,"threads_started":1}
W20260926 23:17:38.806802 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:38.798150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:59457 (local address 127.16.62.126:35669)
0926 23:17:38.798320 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:38.798324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:38.798685 (+ 361us) server_negotiation.cc:407] Connection header received
0926 23:17:38.798809 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:38.798813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:38.798875 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:38.798953 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:38.799676 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.800429 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.801691 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:38.802061 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:38.802147 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:38.806548 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:59457: BlockingRecv error: recv got EOF from 127.16.62.65:59457 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 23:17:39 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:39 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:39.073268 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:39.064587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:51095)
0926 23:17:39.065148 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:39.065167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:39.065315 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:39.065713 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:39.065718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:39.066214 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:39.066451 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:39.066465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.067455 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.067459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:39.068450 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:39.068461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.069293 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.069298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:39.069403 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:39.069975 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:39.069991 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:39.073038 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":304,"threads_started":1}
W20260926 23:17:39.073459 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:39.064688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:51095 (local address 127.16.62.126:35669)
0926 23:17:39.064840 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:39.064845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:39.065273 (+ 428us) server_negotiation.cc:407] Connection header received
0926 23:17:39.065436 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:39.065443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:39.065520 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:39.065641 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:39.066607 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.067338 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.068619 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.069124 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.069233 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:39.073253 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:51095: BlockingRecv error: recv got EOF from 127.16.62.67:51095 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 23:17:39 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:39 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:39.398252 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:39.389484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:43977)
0926 23:17:39.389981 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:39.389996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:39.390100 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:39.390388 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:39.390391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:39.390767 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:39.390908 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:39.390916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.391895 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.391899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:39.392964 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:39.392974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.393830 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.393836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:39.394003 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:39.394635 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:39.394652 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:39.397937 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
W20260926 23:17:39.398357 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:39.389538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:43977 (local address 127.16.62.126:35669)
0926 23:17:39.389706 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:39.389711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:39.390064 (+ 353us) server_negotiation.cc:407] Connection header received
0926 23:17:39.390190 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:39.390194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:39.390257 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:39.390336 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:39.391073 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.391748 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.393135 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.393683 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.393816 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:39.398113 (+ 4297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:43977: BlockingRecv error: recv got EOF from 127.16.62.66:43977 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 23:17:39 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:39 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:39.816347 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:39.807718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:58435)
0926 23:17:39.808232 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:39.808247 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:39.808394 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:39.808703 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:39.808707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:39.809113 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:39.809274 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:39.809282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.810256 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.810268 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:39.811189 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:39.811199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.811891 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.811896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:39.812058 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:39.812632 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:39.812649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:39.815979 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
W20260926 23:17:39.816409 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:39.807786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:58435 (local address 127.16.62.126:35669)
0926 23:17:39.808009 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:39.808017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:39.808331 (+ 314us) server_negotiation.cc:407] Connection header received
0926 23:17:39.808506 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:39.808511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:39.808575 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:39.808682 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:39.809415 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.810103 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.811323 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:39.811744 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:39.811875 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:39.816181 (+ 4306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:58435: BlockingRecv error: recv got EOF from 127.16.62.65:58435 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 23:17:40 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:40 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:40.083155 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:40.074291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:56027)
0926 23:17:40.074836 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:40.074855 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:40.074997 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:40.075341 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:40.075346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:40.075774 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:40.075916 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:40.075924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.076992 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.077001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:40.077957 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:40.077967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.078656 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.078662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:40.078839 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:40.079453 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:40.079470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:40.082840 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":246,"threads_started":1}
W20260926 23:17:40.083269 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:40.074368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:56027 (local address 127.16.62.126:35669)
0926 23:17:40.074624 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:40.074631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:40.074954 (+ 323us) server_negotiation.cc:407] Connection header received
0926 23:17:40.075122 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:40.075135 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:40.075207 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:40.075309 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:40.076059 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.076866 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.078099 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.078495 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.078619 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:40.083018 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:56027: BlockingRecv error: recv got EOF from 127.16.62.67:56027 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 23:17:40 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:40 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:40.412218 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:40.399188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:55451)
0926 23:17:40.399682 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:40.399698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:40.399808 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:40.400084 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:40.400087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:40.400651 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:40.400864 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:40.400877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.401866 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.401871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:40.402830 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:40.402840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.403484 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.403489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:40.403663 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:40.408705 (+ 5042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:40.408734 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:40.411870 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
W20260926 23:17:40.412367 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:40.399428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:55451 (local address 127.16.62.126:35669)
0926 23:17:40.399622 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:40.399628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:40.399769 (+ 141us) server_negotiation.cc:407] Connection header received
0926 23:17:40.399903 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:40.399906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:40.399964 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:40.400042 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:40.401017 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.401732 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.402959 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.403352 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.403478 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:40.412051 (+ 8573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:55451: BlockingRecv error: recv got EOF from 127.16.62.66:55451 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 23:17:40 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:40 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:40.826296 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:40.817433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:53703)
0926 23:17:40.817944 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:40.817960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:40.818073 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:40.818354 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:40.818358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:40.818802 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:40.819056 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:40.819073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.820020 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.820023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:40.820938 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:40.820947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.821531 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.821535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:40.821634 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:40.822270 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:40.822292 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:40.825967 (+ 3675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":304,"threads_started":1}
W20260926 23:17:40.826362 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:40.817601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:53703 (local address 127.16.62.126:35669)
0926 23:17:40.817830 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:40.817836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:40.818037 (+ 201us) server_negotiation.cc:407] Connection header received
0926 23:17:40.818173 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:40.818176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:40.818235 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:40.818313 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:40.819186 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.819917 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.821047 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:40.821415 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:40.821501 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:40.826134 (+ 4633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:53703: BlockingRecv error: recv got EOF from 127.16.62.65:53703 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 23:17:41 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:41 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:41.092763 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:41.084040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:45535)
0926 23:17:41.084602 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:41.084619 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:41.084734 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:41.085015 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:41.085018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:41.085362 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:41.085524 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:41.085532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.086461 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.086464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:41.087432 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:41.087442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.088037 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.088041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:41.088262 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:41.088881 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:41.088899 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:41.092404 (+ 3505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":316,"threads_started":1}
W20260926 23:17:41.092876 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:41.084128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:45535 (local address 127.16.62.126:35669)
0926 23:17:41.084325 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:41.084329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:41.084691 (+ 362us) server_negotiation.cc:407] Connection header received
0926 23:17:41.084835 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:41.084838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:41.084894 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:41.084970 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:41.085652 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.086352 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.087545 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.087909 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.087992 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:41.092621 (+ 4629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:45535: BlockingRecv error: recv got EOF from 127.16.62.67:45535 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 23:17:41 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:41 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:41.422295 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:41.413274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:53183)
0926 23:17:41.413749 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:41.413764 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:41.413911 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:41.414233 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:41.414238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:41.414693 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:41.414876 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:41.414885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.415902 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.415910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:41.416853 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:41.416865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.417624 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.417630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:41.417809 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:41.418454 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:41.418471 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:41.422084 (+ 3613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":269,"threads_started":1}
W20260926 23:17:41.422498 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:41.413340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:53183 (local address 127.16.62.126:35669)
0926 23:17:41.413572 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:41.413581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:41.413845 (+ 264us) server_negotiation.cc:407] Connection header received
0926 23:17:41.414028 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:41.414033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:41.414103 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:41.414205 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:41.415020 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.415776 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.417011 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.417450 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.417596 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:41.422239 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:53183: BlockingRecv error: recv got EOF from 127.16.62.66:53183 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 23:17:41 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:41 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:41.835843 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:41.827164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:56341)
0926 23:17:41.827730 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:41.827746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:41.827863 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:41.828155 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:41.828159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:41.828526 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:41.828706 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:41.828714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.829619 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.829622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:41.830515 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:41.830534 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.831189 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.831193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:41.831309 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:41.831990 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:41.832007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:41.835588 (+ 3581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":300,"threads_started":1}
W20260926 23:17:41.836007 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:41.827182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:56341 (local address 127.16.62.126:35669)
0926 23:17:41.827326 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:41.827330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:41.827814 (+ 484us) server_negotiation.cc:407] Connection header received
0926 23:17:41.827959 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:41.827962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:41.828017 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:41.828092 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:41.828819 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.829505 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.830634 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:41.831038 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:41.831130 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:41.835771 (+ 4641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:56341: BlockingRecv error: recv got EOF from 127.16.62.65:56341 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 23:17:42 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:42 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:42.102303 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:42.093843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:54937)
0926 23:17:42.094286 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:42.094302 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:42.094410 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:42.094673 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:42.094676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:42.095213 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:42.095431 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:42.095440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.096549 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.096552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:42.097437 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:42.097447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.098073 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.098077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:42.098209 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:42.098758 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:42.098774 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:42.101960 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
W20260926 23:17:42.102378 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:42.093904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:54937 (local address 127.16.62.126:35669)
0926 23:17:42.094129 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:42.094135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:42.094377 (+ 242us) server_negotiation.cc:407] Connection header received
0926 23:17:42.094498 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:42.094501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:42.094559 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:42.094636 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:42.095550 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.096437 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.097551 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.097958 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.098044 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:42.102130 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:54937: BlockingRecv error: recv got EOF from 127.16.62.67:54937 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 23:17:42 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:42 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:42.432375 25906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:42.423211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:54705)
0926 23:17:42.423733 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:42.423750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:42.423871 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:42.424205 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:42.424208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:42.424575 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:42.424751 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:42.424758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.426031 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.426035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:42.426963 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:42.426974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.427913 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.427917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:42.428037 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:42.428604 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:42.428621 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:42.431968 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":277,"threads_started":1}
W20260926 23:17:42.432552 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:42.423250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:54705 (local address 127.16.62.126:35669)
0926 23:17:42.423502 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:42.423511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:42.423831 (+ 320us) server_negotiation.cc:407] Connection header received
0926 23:17:42.423989 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:42.423993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:42.424051 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:42.424128 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:42.424882 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.425886 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.427109 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.427759 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.427875 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:42.432268 (+ 4393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:54705: BlockingRecv error: recv got EOF from 127.16.62.66:54705 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 23:17:42 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:42 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:42.845645 25907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:42.836782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:48697)
0926 23:17:42.837245 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:42.837261 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:42.837414 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:42.837743 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:42.837748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:42.838165 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:42.838305 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:42.838314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.839334 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.839340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:42.840381 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:42.840401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.841248 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.841254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:42.841440 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:42.842045 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:42.842063 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:42.845412 (+ 3349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":239,"threads_started":1}
W20260926 23:17:42.845826 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:42.836851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:48697 (local address 127.16.62.126:35669)
0926 23:17:42.837074 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:42.837083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:42.837346 (+ 263us) server_negotiation.cc:407] Connection header received
0926 23:17:42.837535 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:42.837540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:42.837609 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:42.837710 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:42.838445 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.839197 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.840530 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:42.841090 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:42.841227 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:42.845594 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:48697: BlockingRecv error: recv got EOF from 127.16.62.65:48697 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 23:17:43 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:43 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:43.112258 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:43.103186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:48809)
0926 23:17:43.103736 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:43.103751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:43.103914 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:43.104273 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:43.104278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:43.104681 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:43.104841 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:43.104849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.105879 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.105884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:43.106808 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:43.106819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.107540 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.107546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:43.107723 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:43.108376 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:43.108393 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:43.111950 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":307,"threads_started":1}
W20260926 23:17:43.112345 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:43.103310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:48809 (local address 127.16.62.126:35669)
0926 23:17:43.103537 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:43.103544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:43.103838 (+ 294us) server_negotiation.cc:407] Connection header received
0926 23:17:43.104036 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:43.104041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:43.104108 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:43.104234 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:43.105005 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.105747 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.106941 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.107348 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.107505 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:43.112110 (+ 4605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:48809: BlockingRecv error: recv got EOF from 127.16.62.67:48809 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 23:17:43 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:43 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:43.441768 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:43.433385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:44027)
0926 23:17:43.433840 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:43.433855 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:43.433958 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:43.434225 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:43.434228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:43.434597 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:43.434734 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:43.434742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.435687 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.435690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:43.436735 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:43.436747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.437407 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.437411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:43.437521 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:43.438087 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:43.438103 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:43.441537 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":255,"threads_started":1}
W20260926 23:17:43.441931 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:43.433523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:44027 (local address 127.16.62.126:35669)
0926 23:17:43.433701 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:43.433706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:43.433925 (+ 219us) server_negotiation.cc:407] Connection header received
0926 23:17:43.434048 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:43.434052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:43.434105 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:43.434180 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:43.434848 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.435566 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.436857 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.437294 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.437382 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:43.441711 (+ 4329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:44027: BlockingRecv error: recv got EOF from 127.16.62.66:44027 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 23:17:43 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:43 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:43.856468 25910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:43.846861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:45143)
0926 23:17:43.847358 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:43.847375 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:43.847548 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:43.847963 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:43.847970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:43.848405 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:43.848660 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:43.848673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.850078 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.850084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:43.851066 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:43.851077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.851736 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.851741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:43.851875 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:43.852553 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:43.852574 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:43.856186 (+ 3612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":257,"threads_started":1}
W20260926 23:17:43.856590 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:43.846933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:45143 (local address 127.16.62.126:35669)
0926 23:17:43.847104 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:43.847108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:43.847481 (+ 373us) server_negotiation.cc:407] Connection header received
0926 23:17:43.847718 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:43.847729 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:43.847823 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:43.847931 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:43.848821 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.849895 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.851189 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:43.851577 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:43.851676 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:43.856396 (+ 4720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:45143: BlockingRecv error: recv got EOF from 127.16.62.65:45143 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 23:17:44 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:44 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.67@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:44.122151 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:44.113198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.67:54419)
0926 23:17:44.113664 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:44.113679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:44.113787 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:44.114058 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:44.114062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:44.114436 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:44.114590 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:44.114599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.115586 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.115589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:44.116600 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:44.116611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.117237 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.117241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:44.117418 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:44.118053 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:44.118072 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:44.121927 (+ 3855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":248,"threads_started":1}
W20260926 23:17:44.122336 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:44.113218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:54419 (local address 127.16.62.126:35669)
0926 23:17:44.113522 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:44.113530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:44.113761 (+ 231us) server_negotiation.cc:407] Connection header received
0926 23:17:44.113881 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:44.113884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:44.113941 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:44.114018 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:44.114707 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.115454 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.116723 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.117113 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.117213 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:44.122118 (+ 4905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.67:54419: BlockingRecv error: recv got EOF from 127.16.62.67:54419 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Sep 26 23:17:44 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:44 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.66@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:44.451553 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:44.442907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.66:53625)
0926 23:17:44.443424 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:44.443438 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:44.443566 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:44.443903 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:44.443906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:44.444259 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:44.444493 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:44.444502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.445432 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.445435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:44.446318 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:44.446328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.446981 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.446985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:44.447099 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:44.447734 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:44.447750 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:44.451198 (+ 3448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":295,"threads_started":1}
W20260926 23:17:44.451587 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:44.443028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:53625 (local address 127.16.62.126:35669)
0926 23:17:44.443218 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:44.443222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:44.443507 (+ 285us) server_negotiation.cc:407] Connection header received
0926 23:17:44.443671 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:44.443674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:44.443746 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:44.443845 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:44.444610 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.445319 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.446428 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.446844 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.446970 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:44.451373 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.66:53625: BlockingRecv error: recv got EOF from 127.16.62.66:53625 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 23:17:44.860668 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25452
Sep 26 23:17:44 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:17:44 dist-test-slave-vcv1 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.62.65@KRBTEST.COM for oryx/127.16.62.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:17:44.867640 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:44.857392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35669 (local address 127.16.62.65:50017)
0926 23:17:44.857878 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:44.857893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:44.858038 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:44.858432 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:44.858437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:44.858869 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:44.859029 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:44.859037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.860065 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.860071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:44.861154 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:44.861171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.862112 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.862122 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:44.862328 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:44.862975 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:44.862996 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:44.867319 (+ 4323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.62.126:35669: Server oryx/127.16.62.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":270,"threads_started":1}
W20260926 23:17:44.867699 25602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:44.857480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:50017 (local address 127.16.62.126:35669)
0926 23:17:44.857715 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:44.857722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:44.857977 (+ 255us) server_negotiation.cc:407] Connection header received
0926 23:17:44.858186 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:44.858191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:44.858284 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:44.858401 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:44.859170 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.859931 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.861321 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.861951 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.862078 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:44.867496 (+ 5418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.62.65:50017: BlockingRecv error: recv got EOF from 127.16.62.65:50017 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:17:44.874328 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:44.861744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.16.62.65:38667)
0926 23:17:44.862510 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:44.862517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:44.862539 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:17:44.862583 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:44.862589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:44.862816 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:44.862968 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:44.864807 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.866519 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.868268 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.868779 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.868873 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:44.871167 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:44.871175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:44.871187 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:44.871219 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:44.873853 (+ 2634us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.62.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 23:17:44.874106 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39024: Request ticket server kudu/127.16.62.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":454,"threads_started":1}
W20260926 23:17:44.874477 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25452: Not authorized: Client connection negotiation failed: client connection to 127.16.62.65:38667: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.62.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 23:17:44.874687 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 25452
I20260926 23:17:44.892032 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25603
W20260926 23:17:44.905300 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25603: Not authorized: Client connection negotiation failed: client connection to 127.16.62.66:34403: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.62.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 23:17:44.905488 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 25603
I20260926 23:17:44.922691 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25736
I20260926 23:17:44.935961 25917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:44.924437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.16.62.67:35397)
0926 23:17:44.924960 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:44.924970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:44.924999 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:17:44.925074 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:44.925082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:44.925381 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:44.925615 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:44.926791 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.927958 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.929162 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:44.929713 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:44.929808 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:44.932115 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:44.932123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:44.932194 (+ 71us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:44.932248 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:44.935319 (+ 3071us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.62.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 23:17:44.935638 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46714: Request ticket server kudu/127.16.62.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
W20260926 23:17:44.935941 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25736: Not authorized: Client connection negotiation failed: client connection to 127.16.62.67:35397: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.62.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 23:17:44.936050 25917 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46714: Request ticket server kudu/127.16.62.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 23:17:44.936105 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 25736
I20260926 23:17:44.952301 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25383
I20260926 23:17:44.953433 25445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:45.066810 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 25383
2026-09-26T23:17:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21480 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T23:17:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:17:45Z Disabled control of system clock
I20260926 23:17:45.141451 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:36625
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:36625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 23:17:45.453656 25922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:45.453920 25922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:45.454089 25922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:45.461094 25922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:45.461182 25922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:45.461257 25922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:45.461354 25922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:45.461423 25922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:45.470093 25922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:36625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:36625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:45.471287 25922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:45.473006 25922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:45.484473 25930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:45.484524 25928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:45.485097 25927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:45.485384 25922 server_base.cc:1034] running on GCE node
I20260926 23:17:45.486109 25922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:45.487421 25922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:45.488622 25922 hybrid_clock.cc:648] HybridClock initialized: now 1790464665488574 us; error 40 us; skew 500 ppm
I20260926 23:17:45.488978 25922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:17:45.496896 25922 webserver.cc:460] Webserver started at http://127.16.62.126:41605/ using document root <none> and password file <none>
I20260926 23:17:45.497577 25922 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:45.497686 25922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:45.498034 25922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:45.501348 25922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "ad9b1158f6154dd18809d8026123779d"
format_stamp: "Formatted at 2026-09-26 23:17:45 on dist-test-slave-vcv1"
server_key: "4bbbc759bed983dbf88ef61a1e1f6630"
server_key_iv: "1c9c4b5ca08ab3ae79c54d3e16d09cef"
server_key_version: "encryptionkey@0"
I20260926 23:17:45.502086 25922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "ad9b1158f6154dd18809d8026123779d"
format_stamp: "Formatted at 2026-09-26 23:17:45 on dist-test-slave-vcv1"
server_key: "4bbbc759bed983dbf88ef61a1e1f6630"
server_key_iv: "1c9c4b5ca08ab3ae79c54d3e16d09cef"
server_key_version: "encryptionkey@0"
I20260926 23:17:45.507030 25922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:17:45.510350 25936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:45.511620 25922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:17:45.511794 25922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "ad9b1158f6154dd18809d8026123779d"
format_stamp: "Formatted at 2026-09-26 23:17:45 on dist-test-slave-vcv1"
server_key: "4bbbc759bed983dbf88ef61a1e1f6630"
server_key_iv: "1c9c4b5ca08ab3ae79c54d3e16d09cef"
server_key_version: "encryptionkey@0"
I20260926 23:17:45.512055 25922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:45.531730 25922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:45.532552 25922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:45.532819 25922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:45.550326 25922 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:36625
I20260926 23:17:45.550355 25987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:36625 every 8 connection(s)
I20260926 23:17:45.551643 25922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:17:45.555058 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:45.559422 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 25922
I20260926 23:17:45.559640 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:17:45.560297 16633 external_mini_cluster.cc:1428] Setting key 6191ed7394f3a9f1d2a4dc3034354c1a
I20260926 23:17:45.565294 25988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d: Bootstrap starting.
I20260926 23:17:45.568643 25988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:45.569819 25988 log.cc:824] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:45.573050 25988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d: No bootstrap required, opened a new log
I20260926 23:17:45.574321 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:45.562255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.16.62.126:36625)
0926 23:17:45.563317 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:45.563327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:45.563376 (+ 49us) server_negotiation.cc:407] Connection header received
0926 23:17:45.565039 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:45.565061 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:45.565540 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:45.565851 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:45.566464 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:45.567736 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:45.569586 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:45.570379 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:45.570586 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:45.571641 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:45.571650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:45.571666 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:45.572375 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:45.572484 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:45.572509 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:45.572632 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:45.572811 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 23:17:45.573111 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":391,"threads_started":1}
I20260926 23:17:45.581331 25988 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9b1158f6154dd18809d8026123779d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36625 } }
I20260926 23:17:45.581632 25988 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:45.581708 25988 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9b1158f6154dd18809d8026123779d, State: Initialized, Role: FOLLOWER
I20260926 23:17:45.582155 25988 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9b1158f6154dd18809d8026123779d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36625 } }
I20260926 23:17:45.582343 25988 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:17:45.582434 25988 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:17:45.582548 25988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:45.584666 25988 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9b1158f6154dd18809d8026123779d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36625 } }
I20260926 23:17:45.585093 25988 leader_election.cc:302] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad9b1158f6154dd18809d8026123779d; no voters:
I20260926 23:17:45.585632 25988 leader_election.cc:288] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:17:45.585831 25993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:45.586812 25993 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [term 1 LEADER]: Becoming Leader. State: Replica: ad9b1158f6154dd18809d8026123779d, State: Running, Role: LEADER
I20260926 23:17:45.587343 25993 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9b1158f6154dd18809d8026123779d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36625 } }
I20260926 23:17:45.587735 25988 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:45.590261 25994 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad9b1158f6154dd18809d8026123779d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9b1158f6154dd18809d8026123779d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36625 } } }
I20260926 23:17:45.590229 25995 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9b1158f6154dd18809d8026123779d. Latest consensus state: current_term: 1 leader_uuid: "ad9b1158f6154dd18809d8026123779d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9b1158f6154dd18809d8026123779d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36625 } } }
I20260926 23:17:45.590629 25994 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:45.590709 25995 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:45.592268 26000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:17:45.596925 26000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:17:45.604871 26000 catalog_manager.cc:1347] Generated new cluster ID: 31b8db6753614bdd8bee0a1dbed76a8e
I20260926 23:17:45.604972 26000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:17:45.630764 26000 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:17:45.632604 26000 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:17:45.646323 26000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad9b1158f6154dd18809d8026123779d: Generated new TSK 0
I20260926 23:17:45.647173 26000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:17:45.660431 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36625
--builtin_ntp_servers=127.16.62.84:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 23:17:45.988119 26012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:45.988412 26012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:45.988494 26012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:45.995498 26012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:45.995594 26012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:45.995694 26012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:17:46.004312 26012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:46.005505 26012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:46.007190 26012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:46.019745 26020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:46.019886 26017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:46.019843 26018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:46.020879 26012 server_base.cc:1034] running on GCE node
I20260926 23:17:46.021476 26012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:46.022372 26012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:46.023586 26012 hybrid_clock.cc:648] HybridClock initialized: now 1790464666023532 us; error 64 us; skew 500 ppm
I20260926 23:17:46.023903 26012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:17:46.026139 26012 webserver.cc:460] Webserver started at http://127.16.62.65:35497/ using document root <none> and password file <none>
I20260926 23:17:46.026818 26012 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:46.026940 26012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:46.027280 26012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:46.030128 26012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "3802c39a9db248ec9be0218a231617e1"
format_stamp: "Formatted at 2026-09-26 23:17:46 on dist-test-slave-vcv1"
server_key: "da6ef08dcf7da52640f1c5fc95c68abd"
server_key_iv: "d5ae5d79e66dc7cff75b6c556c3e089a"
server_key_version: "encryptionkey@0"
I20260926 23:17:46.030891 26012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "3802c39a9db248ec9be0218a231617e1"
format_stamp: "Formatted at 2026-09-26 23:17:46 on dist-test-slave-vcv1"
server_key: "da6ef08dcf7da52640f1c5fc95c68abd"
server_key_iv: "d5ae5d79e66dc7cff75b6c556c3e089a"
server_key_version: "encryptionkey@0"
I20260926 23:17:46.035836 26012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:17:46.039105 26026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:46.040426 26012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:46.040584 26012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "3802c39a9db248ec9be0218a231617e1"
format_stamp: "Formatted at 2026-09-26 23:17:46 on dist-test-slave-vcv1"
server_key: "da6ef08dcf7da52640f1c5fc95c68abd"
server_key_iv: "d5ae5d79e66dc7cff75b6c556c3e089a"
server_key_version: "encryptionkey@0"
I20260926 23:17:46.040787 26012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:46.080570 26012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:46.081430 26012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:46.081730 26012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:46.082628 26012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:46.084012 26012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:46.084177 26012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:46.084306 26012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:46.084388 26012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:46.116614 26012 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:33301
I20260926 23:17:46.116671 26138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:33301 every 8 connection(s)
I20260926 23:17:46.117892 26012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:17:46.122943 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 26012
I20260926 23:17:46.123154 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:17:46.123909 16633 external_mini_cluster.cc:1428] Setting key f044daa7e5578f0c6adbefd6bfeca097
I20260926 23:17:46.126547 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36625
--builtin_ntp_servers=127.16.62.84:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 23:17:46.130409 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:46.120313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:34147 (local address 127.16.62.126:36625)
0926 23:17:46.120745 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:46.120750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:46.121840 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 23:17:46.124214 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:46.124218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:46.124287 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:46.124380 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:46.125420 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:46.126154 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:46.127717 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:46.128099 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:46.128281 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:46.129600 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:46.129605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:46.129612 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:46.129634 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:46.129685 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:46.129703 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:46.129785 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:46.129992 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 23:17:46.130080 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":239,"threads_started":1}
I20260926 23:17:46.131345 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:46.120868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36625 (local address 127.16.62.65:34147)
0926 23:17:46.121692 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:46.121735 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:46.123872 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:46.124631 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:46.124639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:46.124732 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:46.125255 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:46.125269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:46.126312 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:46.126322 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:46.127570 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:46.127581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:46.128367 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:46.128377 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:46.128608 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:46.129306 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:17:46.129309 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:46.129345 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:17:46.129418 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:46.129782 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:46.129792 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:46.129802 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:17:46.129982 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 23:17:46.130341 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":255,"threads_started":1}
I20260926 23:17:46.132959 26139 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36625
I20260926 23:17:46.133255 26139 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:46.134351 26139 heartbeater.cc:511] Master 127.16.62.126:36625 requested a full tablet report, sending...
I20260926 23:17:46.136480 25953 ts_manager.cc:194] Registered new tserver with Master: 3802c39a9db248ec9be0218a231617e1 (127.16.62.65:33301)
I20260926 23:17:46.137898 25953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.65:34147
W20260926 23:17:46.446506 26143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:46.446769 26143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:46.446916 26143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:46.454119 26143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:46.454208 26143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:46.454416 26143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:17:46.463263 26143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:46.464615 26143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:46.466353 26143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:46.479338 26148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:46.479370 26149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:46.479542 26151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:46.480131 26143 server_base.cc:1034] running on GCE node
I20260926 23:17:46.480780 26143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:46.481673 26143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:46.482888 26143 hybrid_clock.cc:648] HybridClock initialized: now 1790464666482856 us; error 45 us; skew 500 ppm
I20260926 23:17:46.483191 26143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:17:46.485267 26143 webserver.cc:460] Webserver started at http://127.16.62.66:34703/ using document root <none> and password file <none>
I20260926 23:17:46.485911 26143 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:46.486040 26143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:46.486361 26143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:46.489086 26143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "86a02ebc1d7f461898327295e3560d82"
format_stamp: "Formatted at 2026-09-26 23:17:46 on dist-test-slave-vcv1"
server_key: "b128832151e9e733d0ad2b228ae7d31f"
server_key_iv: "8053039867c0a53100615383bbbdbe55"
server_key_version: "encryptionkey@0"
I20260926 23:17:46.489873 26143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "86a02ebc1d7f461898327295e3560d82"
format_stamp: "Formatted at 2026-09-26 23:17:46 on dist-test-slave-vcv1"
server_key: "b128832151e9e733d0ad2b228ae7d31f"
server_key_iv: "8053039867c0a53100615383bbbdbe55"
server_key_version: "encryptionkey@0"
I20260926 23:17:46.494765 26143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:17:46.498026 26157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:46.499212 26143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:17:46.499375 26143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "86a02ebc1d7f461898327295e3560d82"
format_stamp: "Formatted at 2026-09-26 23:17:46 on dist-test-slave-vcv1"
server_key: "b128832151e9e733d0ad2b228ae7d31f"
server_key_iv: "8053039867c0a53100615383bbbdbe55"
server_key_version: "encryptionkey@0"
I20260926 23:17:46.499576 26143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:46.519037 26143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:46.519855 26143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:46.520193 26143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:46.521109 26143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:46.522497 26143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:46.522629 26143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:46.522761 26143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:46.522848 26143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:46.557278 26143 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:43845
I20260926 23:17:46.557367 26269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:43845 every 8 connection(s)
I20260926 23:17:46.558559 26143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:17:46.560645 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 26143
I20260926 23:17:46.560874 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:17:46.561621 16633 external_mini_cluster.cc:1428] Setting key 9b02a90b7bc3cd19fa870108a0cdf935
I20260926 23:17:46.566150 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36625
--builtin_ntp_servers=127.16.62.84:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 23:17:46.578792 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:46.562791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:46115 (local address 127.16.62.126:36625)
0926 23:17:46.563324 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:46.563333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:46.569139 (+ 5806us) server_negotiation.cc:407] Connection header received
0926 23:17:46.571668 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:46.571674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:46.571753 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:46.571817 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:46.573451 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:46.574495 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:46.576093 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:46.576523 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:46.576661 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:46.578136 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:46.578141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:46.578144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:46.578159 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:46.578190 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:46.578201 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:46.578277 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:46.578512 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 23:17:46.578581 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
I20260926 23:17:46.580252 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:46.561745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36625 (local address 127.16.62.66:46115)
0926 23:17:46.568980 (+ 7235us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:46.569031 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:46.571355 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:46.572338 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:46.572349 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:46.572461 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:46.573227 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:46.573254 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:46.574650 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:46.574656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:46.575933 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:46.575945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:46.576754 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:46.576761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:46.577076 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:46.577801 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:17:46.577807 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:46.577857 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:17:46.577956 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:46.578300 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:46.578313 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:46.578324 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:17:46.578497 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 23:17:46.578892 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6691,"thread_start_us":6468,"threads_started":1}
I20260926 23:17:46.581532 26270 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36625
I20260926 23:17:46.581815 26270 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:46.582518 26270 heartbeater.cc:511] Master 127.16.62.126:36625 requested a full tablet report, sending...
I20260926 23:17:46.583844 25953 ts_manager.cc:194] Registered new tserver with Master: 86a02ebc1d7f461898327295e3560d82 (127.16.62.66:43845)
I20260926 23:17:46.584614 25953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.66:46115
W20260926 23:17:46.890182 26273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:46.890468 26273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:46.890632 26273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:46.898044 26273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:46.898187 26273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:46.898362 26273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:17:46.907315 26273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:46.908601 26273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:46.910332 26273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:46.923148 26278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:46.923095 26279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:46.923666 26281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:46.924111 26273 server_base.cc:1034] running on GCE node
I20260926 23:17:46.924952 26273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:46.926249 26273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:46.927430 26273 hybrid_clock.cc:648] HybridClock initialized: now 1790464666927382 us; error 41 us; skew 500 ppm
I20260926 23:17:46.927752 26273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:17:46.929906 26273 webserver.cc:460] Webserver started at http://127.16.62.67:45447/ using document root <none> and password file <none>
I20260926 23:17:46.930565 26273 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:46.930707 26273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:46.931018 26273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:46.933883 26273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "4d01b86c129c42eabe6268c7a95da148"
format_stamp: "Formatted at 2026-09-26 23:17:46 on dist-test-slave-vcv1"
server_key: "4e7dbbd29be3cda626489724f7d5b412"
server_key_iv: "37e19466ac8e436f3b8ace32af362cb9"
server_key_version: "encryptionkey@0"
I20260926 23:17:46.934648 26273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "4d01b86c129c42eabe6268c7a95da148"
format_stamp: "Formatted at 2026-09-26 23:17:46 on dist-test-slave-vcv1"
server_key: "4e7dbbd29be3cda626489724f7d5b412"
server_key_iv: "37e19466ac8e436f3b8ace32af362cb9"
server_key_version: "encryptionkey@0"
I20260926 23:17:46.939486 26273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 23:17:46.942780 26287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:46.944012 26273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:17:46.944180 26273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "4d01b86c129c42eabe6268c7a95da148"
format_stamp: "Formatted at 2026-09-26 23:17:46 on dist-test-slave-vcv1"
server_key: "4e7dbbd29be3cda626489724f7d5b412"
server_key_iv: "37e19466ac8e436f3b8ace32af362cb9"
server_key_version: "encryptionkey@0"
I20260926 23:17:46.944375 26273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:46.972189 26273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:46.972971 26273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:46.973263 26273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:46.974130 26273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:46.975455 26273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:46.975567 26273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:46.975698 26273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:46.975806 26273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:47.008435 26273 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:38197
I20260926 23:17:47.008571 26399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:38197 every 8 connection(s)
I20260926 23:17:47.009722 26273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:17:47.013200 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 26273
I20260926 23:17:47.013427 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:17:47.014015 16633 external_mini_cluster.cc:1428] Setting key 645791f8b1c9e78c0c62bd0eddff9e38
I20260926 23:17:47.023350 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:47.012248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:37631 (local address 127.16.62.126:36625)
0926 23:17:47.012464 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:47.012470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:47.014153 (+ 1683us) server_negotiation.cc:407] Connection header received
0926 23:17:47.016677 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:47.016684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:47.016783 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:47.016902 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:47.018116 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.018871 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.020542 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.020970 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.021118 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:47.022462 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:47.022467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:47.022470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:47.022495 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:47.022536 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:47.022549 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:47.022651 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:47.022885 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 23:17:47.022985 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:17:47.024422 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:47.012819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36625 (local address 127.16.62.67:37631)
0926 23:17:47.013919 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:47.013966 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:47.016382 (+ 2416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:47.017149 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:47.017158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:47.017252 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:47.017930 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:47.017948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.019035 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.019043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:47.020381 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:47.020394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.021201 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.021209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:47.021456 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:47.022130 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:17:47.022133 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:47.022176 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:17:47.022253 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:47.022649 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:47.022662 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:47.022673 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:17:47.022869 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 23:17:47.023249 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":293,"threads_started":1}
I20260926 23:17:47.025704 26400 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36625
I20260926 23:17:47.025992 26400 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:47.026690 26400 heartbeater.cc:511] Master 127.16.62.126:36625 requested a full tablet report, sending...
I20260926 23:17:47.028066 25953 ts_manager.cc:194] Registered new tserver with Master: 4d01b86c129c42eabe6268c7a95da148 (127.16.62.67:38197)
I20260926 23:17:47.029024 25953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.67:37631
I20260926 23:17:47.029765 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 23:17:47.035485 26141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:47.034002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.16.62.126:36625)
0926 23:17:47.034145 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:47.034149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:47.034624 (+ 475us) server_negotiation.cc:407] Connection header received
0926 23:17:47.034726 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:47.034729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:47.034794 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:47.034870 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:47.035243 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37008: BlockingRecv error: recv got EOF from 127.0.0.1:37008 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:17:47.036223 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 26012
I20260926 23:17:47.043520 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:47.037036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.16.62.65:33301)
0926 23:17:47.037764 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:47.037770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:47.037792 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:17:47.037841 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:47.037846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:47.038018 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:47.038132 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:47.038456 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.039367 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.040711 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.041229 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.041324 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:47.042238 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:47.042245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:47.042256 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:47.042803 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:47.042843 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:47.042862 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:47.042945 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:47.043150 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 23:17:47.043246 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":468,"threads_started":1}
I20260926 23:17:47.044751 26134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:47.128319 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 26012
I20260926 23:17:47.144616 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 26143
I20260926 23:17:47.152073 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:47.145544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.16.62.66:43845)
0926 23:17:47.146036 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:47.146043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:47.146069 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:17:47.146126 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:47.146136 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:47.146353 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:47.146477 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:47.146813 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.147898 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.149175 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.149723 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.149820 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:47.150706 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:47.150714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:47.150727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:47.151296 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:47.151340 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:47.151357 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:47.151462 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:47.151591 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 23:17:47.151735 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":265,"threads_started":1}
I20260926 23:17:47.153301 26265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:47.233961 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 26143
I20260926 23:17:47.249682 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 26273
I20260926 23:17:47.257663 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:47.250573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34382 (local address 127.16.62.67:38197)
0926 23:17:47.251090 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:47.251097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:47.251126 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:17:47.251181 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:47.251187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:47.251399 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:47.251566 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:47.252018 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.253080 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.254327 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:47.254887 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:47.255024 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:47.256101 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:47.256110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:47.256123 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:47.256851 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:47.256894 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:47.256913 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:47.257035 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:47.257204 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 23:17:47.257324 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":241,"threads_started":1}
I20260926 23:17:47.258978 26395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:47.342486 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 26273
I20260926 23:17:47.358770 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 25922
I20260926 23:17:47.359922 25983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:47.461162 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 25922
2026-09-26T23:17:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2388 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T23:17:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:17:47Z Disabled control of system clock
I20260926 23:17:47.529142 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41113
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:41113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 23:17:47.848572 26419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:47.848858 26419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:47.848943 26419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:47.855875 26419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:47.855969 26419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:47.856006 26419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:47.856061 26419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:47.856109 26419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:47.864943 26419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:41113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:47.866110 26419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:47.867859 26419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:47.879487 26424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:47.879451 26425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:47.879415 26427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:47.880177 26419 server_base.cc:1034] running on GCE node
I20260926 23:17:47.880951 26419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:47.882247 26419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:47.883474 26419 hybrid_clock.cc:648] HybridClock initialized: now 1790464667883440 us; error 51 us; skew 500 ppm
I20260926 23:17:47.883836 26419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:17:47.886133 26419 webserver.cc:460] Webserver started at http://127.16.62.126:43409/ using document root <none> and password file <none>
I20260926 23:17:47.886809 26419 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:47.886935 26419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:47.887253 26419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:47.890079 26419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "8c791a78a49d4bf4a0adc7273eca48f4"
format_stamp: "Formatted at 2026-09-26 23:17:47 on dist-test-slave-vcv1"
server_key: "2e768c6bad22f229159caf9fba8a2139"
server_key_iv: "da66805d13af598eca4e628efd8e12ba"
server_key_version: "encryptionkey@0"
I20260926 23:17:47.890894 26419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "8c791a78a49d4bf4a0adc7273eca48f4"
format_stamp: "Formatted at 2026-09-26 23:17:47 on dist-test-slave-vcv1"
server_key: "2e768c6bad22f229159caf9fba8a2139"
server_key_iv: "da66805d13af598eca4e628efd8e12ba"
server_key_version: "encryptionkey@0"
I20260926 23:17:47.895861 26419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 23:17:47.899204 26433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:47.900605 26419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 23:17:47.900810 26419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "8c791a78a49d4bf4a0adc7273eca48f4"
format_stamp: "Formatted at 2026-09-26 23:17:47 on dist-test-slave-vcv1"
server_key: "2e768c6bad22f229159caf9fba8a2139"
server_key_iv: "da66805d13af598eca4e628efd8e12ba"
server_key_version: "encryptionkey@0"
I20260926 23:17:47.901022 26419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:47.906412 26419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:47.907160 26419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:47.907455 26419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:47.925243 26419 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:41113
I20260926 23:17:47.925285 26484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:41113 every 8 connection(s)
I20260926 23:17:47.926637 26419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:17:47.928324 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 26419
I20260926 23:17:47.928556 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:17:47.929962 16633 external_mini_cluster.cc:1428] Setting key 045ca6418708d8033fb685b590a00b13
I20260926 23:17:47.931319 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:47.938660 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:47.931961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.16.62.126:41113)
0926 23:17:47.933030 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:47.933040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:47.933078 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:17:47.935363 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:47.935384 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:47.935806 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:47.936123 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:47.937154 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:47.937160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:47.937171 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:47.937184 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:47.937245 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:47.937260 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:47.937429 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:47.937524 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 23:17:47.937812 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":431,"threads_started":1}
I20260926 23:17:47.943367 26485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4: Bootstrap starting.
I20260926 23:17:47.946465 26485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:47.947420 26485 log.cc:824] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:47.949970 26485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4: No bootstrap required, opened a new log
I20260926 23:17:47.956614 26485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c791a78a49d4bf4a0adc7273eca48f4" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41113 } }
I20260926 23:17:47.956934 26485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:47.957041 26485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c791a78a49d4bf4a0adc7273eca48f4, State: Initialized, Role: FOLLOWER
I20260926 23:17:47.957553 26485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c791a78a49d4bf4a0adc7273eca48f4" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41113 } }
I20260926 23:17:47.957774 26485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:17:47.957901 26485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:17:47.958065 26485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:47.960038 26485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c791a78a49d4bf4a0adc7273eca48f4" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41113 } }
I20260926 23:17:47.960529 26485 leader_election.cc:302] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c791a78a49d4bf4a0adc7273eca48f4; no voters:
I20260926 23:17:47.961072 26485 leader_election.cc:288] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:17:47.961208 26490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:47.961941 26490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [term 1 LEADER]: Becoming Leader. State: Replica: 8c791a78a49d4bf4a0adc7273eca48f4, State: Running, Role: LEADER
I20260926 23:17:47.962427 26490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c791a78a49d4bf4a0adc7273eca48f4" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41113 } }
I20260926 23:17:47.963092 26485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:47.965137 26492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c791a78a49d4bf4a0adc7273eca48f4. Latest consensus state: current_term: 1 leader_uuid: "8c791a78a49d4bf4a0adc7273eca48f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c791a78a49d4bf4a0adc7273eca48f4" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41113 } } }
I20260926 23:17:47.965478 26492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:47.965077 26491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c791a78a49d4bf4a0adc7273eca48f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c791a78a49d4bf4a0adc7273eca48f4" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41113 } } }
I20260926 23:17:47.966624 26491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4 [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:47.966789 26499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:17:47.972218 26499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:17:47.980418 26499 catalog_manager.cc:1347] Generated new cluster ID: 17e1117ffbba444694fa205b50ba1bdc
I20260926 23:17:47.980516 26499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:17:48.004725 26499 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:17:48.005112 26499 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:17:48.020215 26499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c791a78a49d4bf4a0adc7273eca48f4: Generated new TSK 0
I20260926 23:17:48.020996 26499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:17:48.035077 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41113
--builtin_ntp_servers=127.16.62.84:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 23:17:48.357967 26509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:48.358220 26509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:48.358304 26509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:48.365311 26509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:48.365396 26509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:48.365500 26509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:17:48.374502 26509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:48.375630 26509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:48.377295 26509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:48.389919 26514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:48.390018 26515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:48.390187 26517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:48.391016 26509 server_base.cc:1034] running on GCE node
I20260926 23:17:48.391649 26509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:48.392598 26509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:48.393792 26509 hybrid_clock.cc:648] HybridClock initialized: now 1790464668393758 us; error 27 us; skew 500 ppm
I20260926 23:17:48.394088 26509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:17:48.396307 26509 webserver.cc:460] Webserver started at http://127.16.62.65:44965/ using document root <none> and password file <none>
I20260926 23:17:48.396937 26509 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:48.397058 26509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:48.397364 26509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:48.400048 26509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "8a1d2a72ae9e46dbaa8860714fedf888"
format_stamp: "Formatted at 2026-09-26 23:17:48 on dist-test-slave-vcv1"
server_key: "a3b718178fbecc2d3d0681a392af2c8a"
server_key_iv: "2e8d032ac5f5a3ef4061f15f227e1cdd"
server_key_version: "encryptionkey@0"
I20260926 23:17:48.400867 26509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "8a1d2a72ae9e46dbaa8860714fedf888"
format_stamp: "Formatted at 2026-09-26 23:17:48 on dist-test-slave-vcv1"
server_key: "a3b718178fbecc2d3d0681a392af2c8a"
server_key_iv: "2e8d032ac5f5a3ef4061f15f227e1cdd"
server_key_version: "encryptionkey@0"
I20260926 23:17:48.405920 26509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:17:48.409152 26523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:48.410392 26509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:48.410533 26509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "8a1d2a72ae9e46dbaa8860714fedf888"
format_stamp: "Formatted at 2026-09-26 23:17:48 on dist-test-slave-vcv1"
server_key: "a3b718178fbecc2d3d0681a392af2c8a"
server_key_iv: "2e8d032ac5f5a3ef4061f15f227e1cdd"
server_key_version: "encryptionkey@0"
I20260926 23:17:48.410732 26509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:48.415920 26509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:48.416674 26509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:48.416885 26509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:48.417646 26509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:48.418860 26509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:48.418942 26509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:48.419015 26509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:48.419068 26509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:48.451577 26509 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:42641
I20260926 23:17:48.451622 26635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:42641 every 8 connection(s)
I20260926 23:17:48.452998 26509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:17:48.457811 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 26509
I20260926 23:17:48.457993 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:17:48.458598 16633 external_mini_cluster.cc:1428] Setting key 899d323da594e607172cab89b88506a0
I20260926 23:17:48.461338 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:48.455445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:36223 (local address 127.16.62.126:41113)
0926 23:17:48.455869 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:48.455874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:48.457141 (+ 1267us) server_negotiation.cc:407] Connection header received
0926 23:17:48.458908 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:48.458914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:48.458978 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:48.459099 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:48.460593 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:48.460597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:48.460601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:48.460615 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:48.460680 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:48.460694 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:48.460803 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:48.460985 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 23:17:48.461044 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":217,"threads_started":1}
I20260926 23:17:48.461683 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41113
--builtin_ntp_servers=127.16.62.84:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 23:17:48.462235 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:48.456069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41113 (local address 127.16.62.65:36223)
0926 23:17:48.456998 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:48.457033 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:48.458614 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:48.459366 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:48.459379 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:48.459501 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:48.460264 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:17:48.460267 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:48.460302 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:17:48.460409 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:48.460791 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:48.460799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:48.460805 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 23:17:48.461017 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 23:17:48.461233 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":270,"threads_started":1}
I20260926 23:17:48.466701 26636 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41113
I20260926 23:17:48.467038 26636 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:48.467595 26636 heartbeater.cc:511] Master 127.16.62.126:41113 requested a full tablet report, sending...
I20260926 23:17:48.469588 26450 ts_manager.cc:194] Registered new tserver with Master: 8a1d2a72ae9e46dbaa8860714fedf888 (127.16.62.65:42641)
W20260926 23:17:48.774174 26640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:48.774441 26640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:48.774609 26640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:48.781826 26640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:48.781929 26640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:48.782143 26640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:17:48.791237 26640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:48.792580 26640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:48.794421 26640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:48.807914 26645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:48.808043 26646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:48.808482 26648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:48.809494 26640 server_base.cc:1034] running on GCE node
I20260926 23:17:48.810052 26640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:48.810918 26640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:48.812167 26640 hybrid_clock.cc:648] HybridClock initialized: now 1790464668812081 us; error 75 us; skew 500 ppm
I20260926 23:17:48.812469 26640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:17:48.814601 26640 webserver.cc:460] Webserver started at http://127.16.62.66:44887/ using document root <none> and password file <none>
I20260926 23:17:48.815248 26640 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:48.815373 26640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:48.815677 26640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:48.818405 26640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "cd5f7b96e9e84ee6b4e1b3f0e152f106"
format_stamp: "Formatted at 2026-09-26 23:17:48 on dist-test-slave-vcv1"
server_key: "8d376164c8b589e0d631da169ccbfcef"
server_key_iv: "62c3396558a778104d2f3f5338a24113"
server_key_version: "encryptionkey@0"
I20260926 23:17:48.819142 26640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "cd5f7b96e9e84ee6b4e1b3f0e152f106"
format_stamp: "Formatted at 2026-09-26 23:17:48 on dist-test-slave-vcv1"
server_key: "8d376164c8b589e0d631da169ccbfcef"
server_key_iv: "62c3396558a778104d2f3f5338a24113"
server_key_version: "encryptionkey@0"
I20260926 23:17:48.823853 26640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260926 23:17:48.826954 26654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:48.828202 26640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 23:17:48.828367 26640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "cd5f7b96e9e84ee6b4e1b3f0e152f106"
format_stamp: "Formatted at 2026-09-26 23:17:48 on dist-test-slave-vcv1"
server_key: "8d376164c8b589e0d631da169ccbfcef"
server_key_iv: "62c3396558a778104d2f3f5338a24113"
server_key_version: "encryptionkey@0"
I20260926 23:17:48.828570 26640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:48.833671 26640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:48.834388 26640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:48.834645 26640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:48.835582 26640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:48.836900 26640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:48.837016 26640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:48.837149 26640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:48.837236 26640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:48.869150 26640 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:44231
I20260926 23:17:48.869247 26766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:44231 every 8 connection(s)
I20260926 23:17:48.870410 26640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:17:48.872383 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 26640
I20260926 23:17:48.872550 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:17:48.873032 16633 external_mini_cluster.cc:1428] Setting key a71d4b4ee29fa3cafc1bf03cb6e1d6c5
I20260926 23:17:48.875805 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41113
--builtin_ntp_servers=127.16.62.84:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 23:17:48.879088 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:48.873240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:45269 (local address 127.16.62.126:41113)
0926 23:17:48.873451 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:48.873458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:48.875004 (+ 1546us) server_negotiation.cc:407] Connection header received
0926 23:17:48.876890 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:48.876895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:48.876970 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:48.877088 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:48.878360 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:48.878366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:48.878370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:48.878388 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:48.878426 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:48.878438 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:48.878556 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:48.878751 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 23:17:48.878815 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:17:48.879931 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:48.873838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41113 (local address 127.16.62.66:45269)
0926 23:17:48.874853 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:48.874907 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:48.876646 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:48.877307 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:48.877314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:48.877399 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:48.878077 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:17:48.878079 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:48.878115 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:17:48.878192 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:48.878527 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:48.878537 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:48.878544 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 23:17:48.878816 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 23:17:48.879044 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":333,"threads_started":1}
I20260926 23:17:48.881279 26767 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41113
I20260926 23:17:48.881549 26767 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:48.882061 26767 heartbeater.cc:511] Master 127.16.62.126:41113 requested a full tablet report, sending...
I20260926 23:17:48.883466 26450 ts_manager.cc:194] Registered new tserver with Master: cd5f7b96e9e84ee6b4e1b3f0e152f106 (127.16.62.66:44231)
W20260926 23:17:49.198546 26770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:49.198820 26770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:49.198915 26770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:49.206136 26770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:49.206239 26770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:49.206351 26770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:17:49.215368 26770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:49.216709 26770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:49.218401 26770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:49.231321 26778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:49.231294 26776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:49.231271 26775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:49.232568 26770 server_base.cc:1034] running on GCE node
I20260926 23:17:49.233136 26770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:49.234011 26770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:49.235239 26770 hybrid_clock.cc:648] HybridClock initialized: now 1790464669235201 us; error 52 us; skew 500 ppm
I20260926 23:17:49.235543 26770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:17:49.237807 26770 webserver.cc:460] Webserver started at http://127.16.62.67:41945/ using document root <none> and password file <none>
I20260926 23:17:49.238452 26770 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:49.238579 26770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:49.238888 26770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:49.241590 26770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "2be0381555a64d8bbb61ec73e9c0392f"
format_stamp: "Formatted at 2026-09-26 23:17:49 on dist-test-slave-vcv1"
server_key: "51c1fdeebeeba712ad72ecf4984d388b"
server_key_iv: "207008345878af5cfb8e9fad192b394e"
server_key_version: "encryptionkey@0"
I20260926 23:17:49.242362 26770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "2be0381555a64d8bbb61ec73e9c0392f"
format_stamp: "Formatted at 2026-09-26 23:17:49 on dist-test-slave-vcv1"
server_key: "51c1fdeebeeba712ad72ecf4984d388b"
server_key_iv: "207008345878af5cfb8e9fad192b394e"
server_key_version: "encryptionkey@0"
I20260926 23:17:49.247037 26770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 23:17:49.250277 26784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:49.251547 26770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:17:49.251718 26770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "2be0381555a64d8bbb61ec73e9c0392f"
format_stamp: "Formatted at 2026-09-26 23:17:49 on dist-test-slave-vcv1"
server_key: "51c1fdeebeeba712ad72ecf4984d388b"
server_key_iv: "207008345878af5cfb8e9fad192b394e"
server_key_version: "encryptionkey@0"
I20260926 23:17:49.251933 26770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:49.257346 26770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:49.258073 26770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:49.258343 26770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:49.259233 26770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:49.260761 26770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:49.260878 26770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:49.261008 26770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:49.261091 26770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:49.294256 26770 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:42451
I20260926 23:17:49.294305 26896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:42451 every 8 connection(s)
I20260926 23:17:49.295585 26770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:17:49.297847 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 26770
I20260926 23:17:49.298019 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:17:49.298592 16633 external_mini_cluster.cc:1428] Setting key 7bebd7c494c18d388758c6deb26712a1
I20260926 23:17:49.304265 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:49.298254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:59329 (local address 127.16.62.126:41113)
0926 23:17:49.298424 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:49.298428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:49.300237 (+ 1809us) server_negotiation.cc:407] Connection header received
0926 23:17:49.302248 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:49.302252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:49.302302 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:49.302380 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:49.303646 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:49.303650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:49.303653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:49.303664 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:49.303689 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:49.303699 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:49.303788 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:49.303945 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 23:17:49.303988 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:17:49.305194 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:49.298925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41113 (local address 127.16.62.67:59329)
0926 23:17:49.300074 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:49.300121 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:49.302004 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:49.302586 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:49.302594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:49.302683 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:49.303380 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:17:49.303382 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:49.303418 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:17:49.303501 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:49.303780 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:49.303786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:49.303791 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:17:49.303990 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 23:17:49.304242 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":349,"threads_started":1}
I20260926 23:17:49.306586 26897 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41113
I20260926 23:17:49.306890 26897 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:49.307451 26897 heartbeater.cc:511] Master 127.16.62.126:41113 requested a full tablet report, sending...
I20260926 23:17:49.308774 26450 ts_manager.cc:194] Registered new tserver with Master: 2be0381555a64d8bbb61ec73e9c0392f (127.16.62.67:42451)
I20260926 23:17:49.314045 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 23:17:49.319887 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:17:49.318494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.16.62.126:41113)
0926 23:17:49.318667 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:49.318671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:49.319204 (+ 533us) server_negotiation.cc:407] Connection header received
0926 23:17:49.319340 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:49.319344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:49.319393 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:49.319476 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:49.319709 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34764: BlockingRecv error: recv got EOF from 127.0.0.1:34764 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:17:49.320485 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 26509
I20260926 23:17:49.323944 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:49.321569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.16.62.65:42641)
0926 23:17:49.322110 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:49.322125 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:49.322149 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:17:49.322203 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:49.322210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:49.322421 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:49.322536 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:49.323392 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:49.323398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:49.323408 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:49.323424 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:49.323461 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:49.323475 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:49.323574 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:49.323657 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 23:17:49.323733 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":294,"threads_started":1}
I20260926 23:17:49.325112 26631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:49.404870 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 26509
I20260926 23:17:49.420892 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 26640
I20260926 23:17:49.424372 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:49.421866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.16.62.66:44231)
0926 23:17:49.422382 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:49.422389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:49.422415 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:17:49.422472 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:49.422479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:49.422719 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:49.422840 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:49.423681 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:49.423687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:49.423698 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:49.423716 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:49.423758 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:49.423772 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:49.423871 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:49.423947 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 23:17:49.424055 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":268,"threads_started":1}
I20260926 23:17:49.425513 26762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:49.505443 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 26640
I20260926 23:17:49.521009 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 26770
I20260926 23:17:49.524791 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:49.522075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.16.62.67:42451)
0926 23:17:49.522706 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:49.522713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:49.522741 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:17:49.522799 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:49.522805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:49.523008 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:49.523135 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:49.523990 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:49.523996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:49.524006 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:17:49.524022 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:49.524086 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:17:49.524100 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:49.524312 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:49.524365 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 23:17:49.524480 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":341,"threads_started":1}
I20260926 23:17:49.525957 26892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:49.606756 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 26770
I20260926 23:17:49.622526 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 26419
I20260926 23:17:49.623693 26480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:49.722962 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 26419
2026-09-26T23:17:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2264 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:17:49 dist-test-slave-vcv1 krb5kdc[26915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:17:49 dist-test-slave-vcv1 krb5kdc[26915](info): set up 2 sockets
Sep 26 23:17:49 dist-test-slave-vcv1 krb5kdc[26915](info): commencing operation
krb5kdc: starting...
W20260926 23:17:50.825984 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:17:50 dist-test-slave-vcv1 krb5kdc[26915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:17:51.042927 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41321
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:41321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:51.372418 26928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:51.372691 26928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:51.372788 26928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:51.379559 26928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:17:51.379676 26928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:51.379724 26928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:51.379772 26928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:17:51.379815 26928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:17:51.388990 26928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:41321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:51.390156 26928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:51.391366 26928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:51.402875 26933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:51.402922 26936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:51.402946 26934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:51.405496 26928 server_base.cc:1034] running on GCE node
I20260926 23:17:51.406224 26928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:51.407514 26928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:51.408737 26928 hybrid_clock.cc:648] HybridClock initialized: now 1790464671408689 us; error 54 us; skew 500 ppm
I20260926 23:17:51.409063 26928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:51 dist-test-slave-vcv1 krb5kdc[26915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464671, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:51.411959 26928 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:17:51.413465 26928 webserver.cc:460] Webserver started at http://127.16.62.126:34101/ using document root <none> and password file <none>
I20260926 23:17:51.414179 26928 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:51.414322 26928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:51.414618 26928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:51.417546 26928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "49eb86360172424fb382417d03e18254"
format_stamp: "Formatted at 2026-09-26 23:17:51 on dist-test-slave-vcv1"
server_key: "8ea7a55409085781cf05f35e82f43898"
server_key_iv: "5ff73bdc999d724ee15967341cc50166"
server_key_version: "encryptionkey@0"
I20260926 23:17:51.418304 26928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "49eb86360172424fb382417d03e18254"
format_stamp: "Formatted at 2026-09-26 23:17:51 on dist-test-slave-vcv1"
server_key: "8ea7a55409085781cf05f35e82f43898"
server_key_iv: "5ff73bdc999d724ee15967341cc50166"
server_key_version: "encryptionkey@0"
I20260926 23:17:51.423105 26928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 23:17:51.426317 26943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:51.427510 26928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 23:17:51.427695 26928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "49eb86360172424fb382417d03e18254"
format_stamp: "Formatted at 2026-09-26 23:17:51 on dist-test-slave-vcv1"
server_key: "8ea7a55409085781cf05f35e82f43898"
server_key_iv: "5ff73bdc999d724ee15967341cc50166"
server_key_version: "encryptionkey@0"
I20260926 23:17:51.427881 26928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:51.443312 26928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:51.447064 26928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:51.447350 26928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:51.465085 26928 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:41321
I20260926 23:17:51.465106 26994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:41321 every 8 connection(s)
I20260926 23:17:51.466313 26928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:17:51.469637 26995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:51.472249 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 26928
I20260926 23:17:51.472432 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:17:51.473079 16633 external_mini_cluster.cc:1428] Setting key a48d8f7e23227dabe52fd974a8de12b2
I20260926 23:17:51.481328 26995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254: Bootstrap starting.
I20260926 23:17:51.484860 26995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:51.486135 26995 log.cc:824] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:17:51 dist-test-slave-vcv1 krb5kdc[26915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:51.489100 26995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254: No bootstrap required, opened a new log
I20260926 23:17:51.495599 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:51.475023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49034 (local address 127.16.62.126:41321)
0926 23:17:51.475790 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:51.475800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:51.475838 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:17:51.477415 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:51.477434 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:51.477814 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:51.478113 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:51.479846 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:51.481202 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:51.483068 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:51.483699 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:51.483824 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:51.488701 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:51.488708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:51.488720 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:51.488751 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:51.491763 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:51.492562 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:51.492567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:51.492572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:51.492734 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:51.493132 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:51.493136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:51.493137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:51.493617 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:51.493864 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:51.494266 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 23:17:51.494605 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":329,"threads_started":1}
I20260926 23:17:51.497592 26995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eb86360172424fb382417d03e18254" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41321 } }
I20260926 23:17:51.497937 26995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:51.498049 26995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49eb86360172424fb382417d03e18254, State: Initialized, Role: FOLLOWER
I20260926 23:17:51.498616 26995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eb86360172424fb382417d03e18254" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41321 } }
I20260926 23:17:51.498836 26995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:17:51.498967 26995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:17:51.499130 26995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:51.501240 26995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eb86360172424fb382417d03e18254" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41321 } }
I20260926 23:17:51.501693 26995 leader_election.cc:302] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49eb86360172424fb382417d03e18254; no voters:
I20260926 23:17:51.502247 26995 leader_election.cc:288] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:17:51.502377 27000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:51.503535 27000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [term 1 LEADER]: Becoming Leader. State: Replica: 49eb86360172424fb382417d03e18254, State: Running, Role: LEADER
I20260926 23:17:51.504077 27000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eb86360172424fb382417d03e18254" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41321 } }
I20260926 23:17:51.504292 26995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:17:51.506950 27002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49eb86360172424fb382417d03e18254. Latest consensus state: current_term: 1 leader_uuid: "49eb86360172424fb382417d03e18254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eb86360172424fb382417d03e18254" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41321 } } }
I20260926 23:17:51.507097 27001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49eb86360172424fb382417d03e18254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49eb86360172424fb382417d03e18254" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41321 } } }
I20260926 23:17:51.507421 27002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:51.507426 27001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254 [sys.catalog]: This master's current role is: LEADER
I20260926 23:17:51.509584 27009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:17:51.514580 27009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:17:51.522172 27009 catalog_manager.cc:1347] Generated new cluster ID: 7440497413db40ee94e8049bdc65eb72
I20260926 23:17:51.522280 27009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:17:51.557391 27009 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:17:51.559280 27009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:17:51.569715 27009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254: Generated new TSK 0
I20260926 23:17:51.570678 27009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:17:51.631381 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41321
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:51.957803 27021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:51.958117 27021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:51.958290 27021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:51.965353 27021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:51.965485 27021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:51.965662 27021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:17:51.974869 27021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:51.976131 27021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:51.977416 27021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:51.990136 27027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:51.990132 27026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:51.990348 27029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:51.990887 27021 server_base.cc:1034] running on GCE node
I20260926 23:17:51.991544 27021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:51.992514 27021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:51.993903 27021 hybrid_clock.cc:648] HybridClock initialized: now 1790464671993706 us; error 30 us; skew 500 ppm
I20260926 23:17:51.994277 27021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:51 dist-test-slave-vcv1 krb5kdc[26915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464671, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:51.997748 27021 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:17:51.999187 27021 webserver.cc:460] Webserver started at http://127.16.62.65:33255/ using document root <none> and password file <none>
I20260926 23:17:51.999820 27021 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:51.999913 27021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:52.000203 27021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:52.003005 27021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9"
format_stamp: "Formatted at 2026-09-26 23:17:51 on dist-test-slave-vcv1"
server_key: "b8d37d71086e2a9f38e7863c05043f74"
server_key_iv: "d744f75e7ac5c3212ddd7da19fcb29d2"
server_key_version: "encryptionkey@0"
I20260926 23:17:52.003649 27021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9"
format_stamp: "Formatted at 2026-09-26 23:17:51 on dist-test-slave-vcv1"
server_key: "b8d37d71086e2a9f38e7863c05043f74"
server_key_iv: "d744f75e7ac5c3212ddd7da19fcb29d2"
server_key_version: "encryptionkey@0"
I20260926 23:17:52.008533 27021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 23:17:52.011698 27036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:52.013123 27021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:17:52.013288 27021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9"
format_stamp: "Formatted at 2026-09-26 23:17:51 on dist-test-slave-vcv1"
server_key: "b8d37d71086e2a9f38e7863c05043f74"
server_key_iv: "d744f75e7ac5c3212ddd7da19fcb29d2"
server_key_version: "encryptionkey@0"
I20260926 23:17:52.013463 27021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:52.024016 27021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:52.028040 27021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:52.028385 27021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:52.029250 27021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:52.030447 27021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:52.030514 27021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:52.030666 27021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:52.030752 27021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:52.062150 27021 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:33549
I20260926 23:17:52.062238 27148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:33549 every 8 connection(s)
I20260926 23:17:52.063474 27021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:17:52.064077 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 27021
I20260926 23:17:52.064378 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:17:52.065174 16633 external_mini_cluster.cc:1428] Setting key 92f9575b224400b512cdac162f2e155e
Sep 26 23:17:52 dist-test-slave-vcv1 krb5kdc[26915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464671, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:52.083822 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:52.066324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:36077 (local address 127.16.62.126:41321)
0926 23:17:52.066743 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:52.066749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:52.067774 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 23:17:52.069685 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:52.069690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:52.069785 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:52.069923 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:52.071321 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:52.072014 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:52.073495 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:52.073885 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:52.074010 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:52.078052 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:52.078057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:52.078061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:52.078089 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:52.080661 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:52.081344 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:52.081356 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:52.081359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:52.081471 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:52.081881 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:52.081885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:52.081889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:52.082146 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:52.082294 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:52.083086 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 23:17:52.083342 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260926 23:17:52.084456 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:52.066888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41321 (local address 127.16.62.65:36077)
0926 23:17:52.067630 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:52.067671 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:52.069413 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:52.070155 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:52.070163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:52.070630 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:52.071160 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:52.071176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:52.072175 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:52.072183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:52.073352 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:52.073363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:52.074091 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:52.074098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:52.074317 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:52.074994 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:52.075014 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:52.077848 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:52.080803 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:52.080811 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:52.080826 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:52.081215 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:52.081575 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:52.081580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:52.081584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:52.081765 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:52.082275 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:52.082282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:52.082619 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 23:17:52.082941 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 23:17:52.083369 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":332,"threads_started":1}
I20260926 23:17:52.086007 27149 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41321
I20260926 23:17:52.086318 27149 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
I20260926 23:17:52.087057 27149 heartbeater.cc:511] Master 127.16.62.126:41321 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
I20260926 23:17:52.089291 26960 ts_manager.cc:194] Registered new tserver with Master: 2eb23ce2ec574f2eb01ecfa2b132a8e9 (127.16.62.65:33549)
I20260926 23:17:52.090744 26960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:36077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:17:52.116482 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41321
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:52.434684 27155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:52.434938 27155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:52.435053 27155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:52.442003 27155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:52.442098 27155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:52.442212 27155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:17:52.451730 27155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:52.452891 27155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:52.454070 27155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:52.466750 27160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:52.466760 27161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:52.467425 27155 server_base.cc:1034] running on GCE node
W20260926 23:17:52.467586 27163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:52.468333 27155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:52.469350 27155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:52.470580 27155 hybrid_clock.cc:648] HybridClock initialized: now 1790464672470533 us; error 42 us; skew 500 ppm
I20260926 23:17:52.470909 27155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:52 dist-test-slave-vcv1 krb5kdc[26915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464672, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:52.474375 27155 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:17:52.475884 27155 webserver.cc:460] Webserver started at http://127.16.62.66:35817/ using document root <none> and password file <none>
I20260926 23:17:52.476605 27155 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:52.476754 27155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:52.477043 27155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:52.479835 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "b40bb249bfa2421c9764c8395b44a9a9"
format_stamp: "Formatted at 2026-09-26 23:17:52 on dist-test-slave-vcv1"
server_key: "adb9343d9d03a106fc264eaffd8982ae"
server_key_iv: "3c5d7d3c2e7cf4f96da0aa2bcc2b3142"
server_key_version: "encryptionkey@0"
I20260926 23:17:52.480597 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "b40bb249bfa2421c9764c8395b44a9a9"
format_stamp: "Formatted at 2026-09-26 23:17:52 on dist-test-slave-vcv1"
server_key: "adb9343d9d03a106fc264eaffd8982ae"
server_key_iv: "3c5d7d3c2e7cf4f96da0aa2bcc2b3142"
server_key_version: "encryptionkey@0"
I20260926 23:17:52.485494 27155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 23:17:52.488839 27170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:52.490057 27155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:17:52.490245 27155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "b40bb249bfa2421c9764c8395b44a9a9"
format_stamp: "Formatted at 2026-09-26 23:17:52 on dist-test-slave-vcv1"
server_key: "adb9343d9d03a106fc264eaffd8982ae"
server_key_iv: "3c5d7d3c2e7cf4f96da0aa2bcc2b3142"
server_key_version: "encryptionkey@0"
I20260926 23:17:52.490429 27155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:52.508517 27155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:52.512413 27155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:52.512693 27155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:52.513628 27155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:52.514886 27155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:52.514987 27155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:52.515151 27155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:52.515224 27155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:52.547721 27155 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:36577
I20260926 23:17:52.547770 27282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:36577 every 8 connection(s)
I20260926 23:17:52.548971 27155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:17:52.557219 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 27155
I20260926 23:17:52.557410 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:17:52.557933 16633 external_mini_cluster.cc:1428] Setting key 87931e17b7298b2cd60c6485d7a3a884
Sep 26 23:17:52 dist-test-slave-vcv1 krb5kdc[26915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464672, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:52.569380 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:52.551339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:53801 (local address 127.16.62.126:41321)
0926 23:17:52.551552 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:52.551558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:52.552689 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 23:17:52.554339 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:52.554343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:52.554421 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:52.554505 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:52.555839 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:52.556627 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:52.558214 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:52.558771 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:52.558908 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:52.562633 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:52.562641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:52.562649 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:52.562700 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:52.566230 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:52.567047 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:52.567055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:52.567060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:52.567209 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:52.567647 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:52.567654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:52.567660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:52.568001 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:52.568109 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:52.568957 (+ 848us) server_negotiation.cc:299] Negotiation successful
0926 23:17:52.569140 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:17:52.570822 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:52.551823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41321 (local address 127.16.62.66:53801)
0926 23:17:52.552534 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:52.552573 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:52.554100 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:52.554692 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:52.554699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:52.555153 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:52.555678 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:52.555693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:52.556757 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:52.556760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:52.558061 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:52.558077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:52.559005 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:52.559011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:52.559280 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:52.560021 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:52.560057 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:52.562413 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:52.566369 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:52.566379 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:52.566394 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:52.566922 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:52.567319 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:52.567326 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:52.567332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:52.567535 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:52.568126 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:52.568156 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:52.568573 (+ 417us) client_negotiation.cc:769] Sending connection context
0926 23:17:52.568905 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 23:17:52.569449 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":324,"threads_started":1}
I20260926 23:17:52.572289 27283 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41321
I20260926 23:17:52.572578 27283 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:52.573261 27283 heartbeater.cc:511] Master 127.16.62.126:41321 requested a full tablet report, sending...
I20260926 23:17:52.574663 26960 ts_manager.cc:194] Registered new tserver with Master: b40bb249bfa2421c9764c8395b44a9a9 (127.16.62.66:36577)
I20260926 23:17:52.575901 26960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:53801
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:17:52.609000 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41321
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:17:52.921381 27288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:17:52.921631 27288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:17:52.921778 27288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:17:52.928486 27288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:17:52.928575 27288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:17:52.928792 27288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:17:52.937983 27288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:17:52.939095 27288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:17:52.940335 27288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:17:52.953778 27294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:52.953969 27296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:17:52.954064 27293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:17:52.954548 27288 server_base.cc:1034] running on GCE node
I20260926 23:17:52.955142 27288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:17:52.956015 27288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:17:52.957257 27288 hybrid_clock.cc:648] HybridClock initialized: now 1790464672957235 us; error 62 us; skew 500 ppm
I20260926 23:17:52.957542 27288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:17:52 dist-test-slave-vcv1 krb5kdc[26915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464672, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:17:52.960493 27288 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:17:52.961736 27288 webserver.cc:460] Webserver started at http://127.16.62.67:42261/ using document root <none> and password file <none>
I20260926 23:17:52.962368 27288 fs_manager.cc:359] Metadata directory not provided
I20260926 23:17:52.962504 27288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:17:52.962827 27288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:17:52.965664 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "9bfdf379050b4f19a9ee6ff5da08400f"
format_stamp: "Formatted at 2026-09-26 23:17:52 on dist-test-slave-vcv1"
server_key: "c8f753b803f87b767317b72429fa8770"
server_key_iv: "14eddcdead36f928083ad06ceeb47474"
server_key_version: "encryptionkey@0"
I20260926 23:17:52.966418 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "9bfdf379050b4f19a9ee6ff5da08400f"
format_stamp: "Formatted at 2026-09-26 23:17:52 on dist-test-slave-vcv1"
server_key: "c8f753b803f87b767317b72429fa8770"
server_key_iv: "14eddcdead36f928083ad06ceeb47474"
server_key_version: "encryptionkey@0"
I20260926 23:17:52.971202 27288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 23:17:52.974579 27303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:52.975896 27288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:17:52.976048 27288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "9bfdf379050b4f19a9ee6ff5da08400f"
format_stamp: "Formatted at 2026-09-26 23:17:52 on dist-test-slave-vcv1"
server_key: "c8f753b803f87b767317b72429fa8770"
server_key_iv: "14eddcdead36f928083ad06ceeb47474"
server_key_version: "encryptionkey@0"
I20260926 23:17:52.976262 27288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:17:52.999894 27288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:17:53.003633 27288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:17:53.003937 27288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:17:53.004891 27288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:17:53.006163 27288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:17:53.006245 27288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:53.006418 27288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:17:53.006469 27288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:17:53.038729 27288 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:34815
I20260926 23:17:53.038815 27415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:34815 every 8 connection(s)
I20260926 23:17:53.039958 27288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:17:53.048547 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 27288
I20260926 23:17:53.048729 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:17:53.049294 16633 external_mini_cluster.cc:1428] Setting key e2dd799229d2515c593d9d0e03d0ad5a
Sep 26 23:17:53 dist-test-slave-vcv1 krb5kdc[26915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464672, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:17:53.059247 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.042332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:35445 (local address 127.16.62.126:41321)
0926 23:17:53.042495 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.042499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.043711 (+ 1212us) server_negotiation.cc:407] Connection header received
0926 23:17:53.045460 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.045465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.045552 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.045685 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:53.047069 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.047812 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.050024 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.050523 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.050675 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.053961 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:53.053974 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:53.053978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:53.054007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:53.056430 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.057105 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.057117 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.057119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.057230 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.057648 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.057656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.057659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.057900 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:53.058057 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.058691 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.058896 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:17:53.060070 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.042829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41321 (local address 127.16.62.67:35445)
0926 23:17:53.043473 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:53.043512 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:53.045204 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:53.045895 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:53.045904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:53.046385 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:17:53.046906 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.046922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.047952 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.047958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.049884 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.049896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.050752 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.050763 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.050994 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.051652 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:17:53.051673 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:17:53.053779 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:17:53.056560 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:53.056568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:53.056582 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:53.056976 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:53.057338 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:17:53.057344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:17:53.057348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:17:53.057516 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:17:53.058034 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:17:53.058041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:17:53.058342 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 23:17:53.058589 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 23:17:53.059053 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":250,"threads_started":1}
I20260926 23:17:53.061280 27416 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41321
I20260926 23:17:53.061520 27416 heartbeater.cc:464] Registering TS with master...
I20260926 23:17:53.062372 27416 heartbeater.cc:511] Master 127.16.62.126:41321 requested a full tablet report, sending...
I20260926 23:17:53.063934 26960 ts_manager.cc:194] Registered new tserver with Master: 9bfdf379050b4f19a9ee6ff5da08400f (127.16.62.67:34815)
I20260926 23:17:53.064497 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:17:53.065011 26960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:35445
I20260926 23:17:53.080636 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.069132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.16.62.126:41321)
0926 23:17:53.069280 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.069284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.069843 (+ 559us) server_negotiation.cc:407] Connection header received
0926 23:17:53.069966 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.069970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.070020 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.070097 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:53.071451 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.072187 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.073386 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.073768 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.073853 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.075901 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:53.075905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:53.075908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:53.075935 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:53.078388 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.078953 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.078957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.078959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.079069 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.079400 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.079403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.079405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.079678 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:53.079863 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.080225 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.080409 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:17:53.084364 26960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:17:53.091061 26960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:17:53.093386 27149 heartbeater.cc:503] Master 127.16.62.126:41321 was elected leader, sending a full tablet report...
I20260926 23:17:53.117193 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.103040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.16.62.66:36577)
0926 23:17:53.103644 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.103656 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.105071 (+ 1415us) server_negotiation.cc:407] Connection header received
0926 23:17:53.105292 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.105302 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.105525 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.105694 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:53.107273 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.109247 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.113004 (+ 3757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.114933 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.115036 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.115260 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.116685 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.116869 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":310,"threads_started":1}
I20260926 23:17:53.117369 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.103017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:34815 (local address 127.0.0.1:50798)
0926 23:17:53.104322 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:53.104354 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:53.104736 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:53.105992 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:53.106004 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:53.106044 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:53.106781 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.106797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.109575 (+ 2778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.109583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.113273 (+ 3690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.113293 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.116191 (+ 2898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.116201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.116460 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.116518 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 23:17:53.116639 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 23:17:53.116839 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":608,"thread_start_us":1756,"threads_started":1}
I20260926 23:17:53.117693 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.103550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.16.62.67:34815)
0926 23:17:53.104907 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.104917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.104949 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:17:53.105047 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.105057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.105326 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.105611 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:53.107061 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.109370 (+ 2309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.113524 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.115533 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.115706 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.116008 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.116780 (+ 772us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.116983 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1199,"thread_start_us":1053,"threads_started":1}
I20260926 23:17:53.117945 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.103047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36577 (local address 127.0.0.1:55446)
0926 23:17:53.104970 (+ 1923us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:53.104983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:53.105104 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:53.106017 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:53.106024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:53.106059 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:53.107057 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.107071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.109482 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.109491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.112779 (+ 3288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.112800 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.115474 (+ 2674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.115483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.115791 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.115863 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 23:17:53.117405 (+ 1542us) client_negotiation.cc:240] Negotiation successful
0926 23:17:53.117529 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1780,"mutex_wait_us":243,"thread_start_us":862,"threads_started":1}
I20260926 23:17:53.121374 27351 tablet_service.cc:1467] Processing CreateTablet for tablet fc08d6d66a474c0d9d2bc855b7852ca3 (DEFAULT_TABLE table=test-table [id=8179fdf6bbbf4a92b290b5d97be630a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:53.121490 27218 tablet_service.cc:1467] Processing CreateTablet for tablet fc08d6d66a474c0d9d2bc855b7852ca3 (DEFAULT_TABLE table=test-table [id=8179fdf6bbbf4a92b290b5d97be630a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:53.122637 27351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc08d6d66a474c0d9d2bc855b7852ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:53.122771 27218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc08d6d66a474c0d9d2bc855b7852ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:53.124645 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.105162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:33549 (local address 127.0.0.1:45326)
0926 23:17:53.116870 (+ 11708us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:53.116889 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:53.117006 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:53.117839 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:53.117843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:53.117869 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:53.118294 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.118305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.120099 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.120104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.122081 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.122094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.123834 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.123841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.124006 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.124037 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:17:53.124192 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 23:17:53.124349 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11567,"thread_start_us":11400,"threads_started":1}
I20260926 23:17:53.124703 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.105489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.16.62.65:33549)
0926 23:17:53.116541 (+ 11052us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.116551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.117129 (+ 578us) server_negotiation.cc:407] Connection header received
0926 23:17:53.117229 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.117239 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.117645 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.117851 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:53.118469 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.119938 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.122263 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.123595 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.123767 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.123971 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.124227 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.124390 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10868,"thread_start_us":10705,"threads_started":1}
I20260926 23:17:53.127957 27084 tablet_service.cc:1467] Processing CreateTablet for tablet fc08d6d66a474c0d9d2bc855b7852ca3 (DEFAULT_TABLE table=test-table [id=8179fdf6bbbf4a92b290b5d97be630a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:17:53.129338 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc08d6d66a474c0d9d2bc855b7852ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:17:53.134388 27434 tablet_bootstrap.cc:492] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: Bootstrap starting.
I20260926 23:17:53.137415 27434 tablet_bootstrap.cc:654] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:53.138388 27434 log.cc:824] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:53.139842 27436 tablet_bootstrap.cc:492] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: Bootstrap starting.
I20260926 23:17:53.141418 27434 tablet_bootstrap.cc:492] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: No bootstrap required, opened a new log
I20260926 23:17:53.141666 27434 ts_tablet_manager.cc:1397] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260926 23:17:53.143085 27437 tablet_bootstrap.cc:492] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: Bootstrap starting.
I20260926 23:17:53.143852 27436 tablet_bootstrap.cc:654] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:53.145010 27436 log.cc:824] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:53.145843 27437 tablet_bootstrap.cc:654] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: Neither blocks nor log segments found. Creating new log.
I20260926 23:17:53.146705 27437 log.cc:824] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: Log is configured to *not* fsync() on all Append() calls
I20260926 23:17:53.153753 27434 raft_consensus.cc:348] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } }
I20260926 23:17:53.154177 27434 raft_consensus.cc:374] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:53.154281 27434 raft_consensus.cc:729] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40bb249bfa2421c9764c8395b44a9a9, State: Initialized, Role: FOLLOWER
I20260926 23:17:53.154315 27437 tablet_bootstrap.cc:492] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: No bootstrap required, opened a new log
I20260926 23:17:53.154498 27436 tablet_bootstrap.cc:492] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: No bootstrap required, opened a new log
I20260926 23:17:53.154593 27437 ts_tablet_manager.cc:1397] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 23:17:53.154729 27436 ts_tablet_manager.cc:1397] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260926 23:17:53.154851 27434 consensus_queue.cc:260] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } }
I20260926 23:17:53.155947 27283 heartbeater.cc:503] Master 127.16.62.126:41321 was elected leader, sending a full tablet report...
I20260926 23:17:53.156584 27434 ts_tablet_manager.cc:1428] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260926 23:17:53.163089 27436 raft_consensus.cc:348] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } }
I20260926 23:17:53.163411 27436 raft_consensus.cc:374] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:53.163517 27436 raft_consensus.cc:729] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfdf379050b4f19a9ee6ff5da08400f, State: Initialized, Role: FOLLOWER
I20260926 23:17:53.163661 27437 raft_consensus.cc:348] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } }
I20260926 23:17:53.164022 27437 raft_consensus.cc:374] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:17:53.164165 27437 raft_consensus.cc:729] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb23ce2ec574f2eb01ecfa2b132a8e9, State: Initialized, Role: FOLLOWER
I20260926 23:17:53.164204 27436 consensus_queue.cc:260] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } }
I20260926 23:17:53.164736 27437 consensus_queue.cc:260] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } }
I20260926 23:17:53.165575 27416 heartbeater.cc:503] Master 127.16.62.126:41321 was elected leader, sending a full tablet report...
I20260926 23:17:53.166128 27436 ts_tablet_manager.cc:1428] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 23:17:53.168710 27437 ts_tablet_manager.cc:1428] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260926 23:17:53.203406 27442 raft_consensus.cc:482] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:17:53.203650 27442 raft_consensus.cc:504] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } }
I20260926 23:17:53.205403 27442 leader_election.cc:288] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40bb249bfa2421c9764c8395b44a9a9 (127.16.62.66:36577), 9bfdf379050b4f19a9ee6ff5da08400f (127.16.62.67:34815)
I20260926 23:17:53.214282 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.205962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:50425 (local address 127.16.62.67:34815)
0926 23:17:53.206173 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.206181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.206824 (+ 643us) server_negotiation.cc:407] Connection header received
0926 23:17:53.206937 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.206942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.207014 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.207100 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:53.207874 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.209007 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.211972 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.213234 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.213396 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.213457 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.213911 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.214028 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:17:53.214282 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.205949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:38093 (local address 127.16.62.66:36577)
0926 23:17:53.206179 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.206185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.206685 (+ 500us) server_negotiation.cc:407] Connection header received
0926 23:17:53.206892 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.206900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.206975 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.207074 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:17:53.207825 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.208966 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.211972 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.213248 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.213389 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.213457 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.213911 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.214028 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 23:17:53.214514 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.205758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:34815 (local address 127.16.62.65:50425)
0926 23:17:53.206712 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:53.206727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:53.206834 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:53.207118 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:53.207137 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:53.207216 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:53.207707 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.207720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.209115 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.209118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.211811 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.211832 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.213515 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.213520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.213722 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.213783 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 23:17:53.213874 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 23:17:53.214034 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":192,"spinlock_wait_cycles":19840,"thread_start_us":382,"threads_started":1}
I20260926 23:17:53.214807 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.205659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36577 (local address 127.16.62.65:38093)
0926 23:17:53.206556 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 23:17:53.206581 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:17:53.206749 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:17:53.207118 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:17:53.207122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:17:53.207150 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:17:53.207650 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.207670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.209100 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.209107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.211834 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:17:53.211842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.213501 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.213508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:17:53.213727 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.213780 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 23:17:53.213874 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 23:17:53.214034 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":309,"thread_start_us":763,"threads_started":1}
I20260926 23:17:53.215510 27371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc08d6d66a474c0d9d2bc855b7852ca3" candidate_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" is_pre_election: true
I20260926 23:17:53.215754 27238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc08d6d66a474c0d9d2bc855b7852ca3" candidate_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40bb249bfa2421c9764c8395b44a9a9" is_pre_election: true
I20260926 23:17:53.216015 27371 raft_consensus.cc:2454] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eb23ce2ec574f2eb01ecfa2b132a8e9 in term 0.
I20260926 23:17:53.216295 27238 raft_consensus.cc:2454] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eb23ce2ec574f2eb01ecfa2b132a8e9 in term 0.
I20260926 23:17:53.216732 27038 leader_election.cc:302] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb23ce2ec574f2eb01ecfa2b132a8e9, 9bfdf379050b4f19a9ee6ff5da08400f; no voters:
I20260926 23:17:53.217085 27442 raft_consensus.cc:2781] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:17:53.217252 27442 raft_consensus.cc:482] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:17:53.217370 27442 raft_consensus.cc:3037] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:53.219461 27442 raft_consensus.cc:504] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } }
I20260926 23:17:53.220038 27442 leader_election.cc:288] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [CANDIDATE]: Term 1 election: Requested vote from peers b40bb249bfa2421c9764c8395b44a9a9 (127.16.62.66:36577), 9bfdf379050b4f19a9ee6ff5da08400f (127.16.62.67:34815)
I20260926 23:17:53.220471 27238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc08d6d66a474c0d9d2bc855b7852ca3" candidate_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40bb249bfa2421c9764c8395b44a9a9"
I20260926 23:17:53.220471 27371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc08d6d66a474c0d9d2bc855b7852ca3" candidate_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfdf379050b4f19a9ee6ff5da08400f"
I20260926 23:17:53.220676 27371 raft_consensus.cc:3037] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:53.220667 27238 raft_consensus.cc:3037] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:17:53.223800 27371 raft_consensus.cc:2454] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eb23ce2ec574f2eb01ecfa2b132a8e9 in term 1.
I20260926 23:17:53.223802 27238 raft_consensus.cc:2454] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eb23ce2ec574f2eb01ecfa2b132a8e9 in term 1.
I20260926 23:17:53.224213 27038 leader_election.cc:302] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb23ce2ec574f2eb01ecfa2b132a8e9, 9bfdf379050b4f19a9ee6ff5da08400f; no voters:
I20260926 23:17:53.224540 27442 raft_consensus.cc:2781] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:17:53.225059 27442 raft_consensus.cc:686] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 1 LEADER]: Becoming Leader. State: Replica: 2eb23ce2ec574f2eb01ecfa2b132a8e9, State: Running, Role: LEADER
I20260926 23:17:53.225508 27442 consensus_queue.cc:237] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } }
I20260926 23:17:53.229796 26959 catalog_manager.cc:5495] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eb23ce2ec574f2eb01ecfa2b132a8e9 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eb23ce2ec574f2eb01ecfa2b132a8e9" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 33549 } health_report { overall_health: HEALTHY } } }
I20260926 23:17:53.267243 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.261207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.16.62.65:33549)
0926 23:17:53.261455 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.261463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.261667 (+ 204us) server_negotiation.cc:407] Connection header received
0926 23:17:53.261944 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.261948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.262036 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.262195 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:17:53.262626 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.263363 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.264882 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.265356 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.265501 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.266033 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:17:53.266582 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:17:53.266744 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.266847 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.266934 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 23:17:53.275418 27238 raft_consensus.cc:1261] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [term 1 FOLLOWER]: Refusing update from remote peer 2eb23ce2ec574f2eb01ecfa2b132a8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:53.275418 27371 raft_consensus.cc:1261] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [term 1 FOLLOWER]: Refusing update from remote peer 2eb23ce2ec574f2eb01ecfa2b132a8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:17:53.276525 27445 consensus_queue.cc:1035] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40bb249bfa2421c9764c8395b44a9a9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:53.276933 27442 consensus_queue.cc:1035] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfdf379050b4f19a9ee6ff5da08400f" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 34815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:17:53.289441 27450 mvcc.cc:205] Tried to move back new op lower bound from 7333743301725396992 to 7333743301535100928. Current Snapshot: MvccSnapshot[applied={T|T < 7333743301725396992}]
I20260926 23:17:53.290880 27449 mvcc.cc:205] Tried to move back new op lower bound from 7333743301725396992 to 7333743301535100928. Current Snapshot: MvccSnapshot[applied={T|T < 7333743301725396992}]
I20260926 23:17:53.300601 26960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:53.300944 26960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:17:53.304546 26960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49eb86360172424fb382417d03e18254: Sending DeleteTablet for 3 replicas of tablet fc08d6d66a474c0d9d2bc855b7852ca3
I20260926 23:17:53.305559 27218 tablet_service.cc:1514] Processing DeleteTablet for tablet fc08d6d66a474c0d9d2bc855b7852ca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:53 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:55446
I20260926 23:17:53.305855 27351 tablet_service.cc:1514] Processing DeleteTablet for tablet fc08d6d66a474c0d9d2bc855b7852ca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:53 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:50798
I20260926 23:17:53.305867 27084 tablet_service.cc:1514] Processing DeleteTablet for tablet fc08d6d66a474c0d9d2bc855b7852ca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:17:53 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:45326
I20260926 23:17:53.308116 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 27021
I20260926 23:17:53.317471 27457 tablet_replica.cc:330] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: stopping tablet replica
I20260926 23:17:53.318287 27457 raft_consensus.cc:2229] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:53.319540 27459 tablet_replica.cc:330] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: stopping tablet replica
I20260926 23:17:53.320016 27459 raft_consensus.cc:2229] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:17:53.320576 27459 raft_consensus.cc:2258] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:53.320106 27457 raft_consensus.cc:2258] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:53.321969 27458 tablet_replica.cc:330] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: stopping tablet replica
I20260926 23:17:53.322355 27458 raft_consensus.cc:2229] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:17:53.323050 27458 raft_consensus.cc:2258] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:17:53.323864 27459 ts_tablet_manager.cc:1905] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:53.324851 27458 ts_tablet_manager.cc:1905] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:53.325340 27457 ts_tablet_manager.cc:1905] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:17:53.330718 27459 ts_tablet_manager.cc:1918] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:53.330884 27459 log.cc:1196] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/fc08d6d66a474c0d9d2bc855b7852ca3
I20260926 23:17:53.331447 27459 ts_tablet_manager.cc:1939] T fc08d6d66a474c0d9d2bc855b7852ca3 P 9bfdf379050b4f19a9ee6ff5da08400f: Deleting consensus metadata
I20260926 23:17:53.333336 26945 catalog_manager.cc:4841] TS 9bfdf379050b4f19a9ee6ff5da08400f (127.16.62.67:34815): tablet fc08d6d66a474c0d9d2bc855b7852ca3 (table test-table [id=8179fdf6bbbf4a92b290b5d97be630a5]) successfully deleted
I20260926 23:17:53.333412 27457 ts_tablet_manager.cc:1918] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:53.333491 27458 ts_tablet_manager.cc:1918] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:17:53.333552 27457 log.cc:1196] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/fc08d6d66a474c0d9d2bc855b7852ca3
I20260926 23:17:53.334075 27457 ts_tablet_manager.cc:1939] T fc08d6d66a474c0d9d2bc855b7852ca3 P b40bb249bfa2421c9764c8395b44a9a9: Deleting consensus metadata
I20260926 23:17:53.335294 27458 log.cc:1196] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/fc08d6d66a474c0d9d2bc855b7852ca3
I20260926 23:17:53.335860 27458 ts_tablet_manager.cc:1939] T fc08d6d66a474c0d9d2bc855b7852ca3 P 2eb23ce2ec574f2eb01ecfa2b132a8e9: Deleting consensus metadata
I20260926 23:17:53.335862 26947 catalog_manager.cc:4841] TS b40bb249bfa2421c9764c8395b44a9a9 (127.16.62.66:36577): tablet fc08d6d66a474c0d9d2bc855b7852ca3 (table test-table [id=8179fdf6bbbf4a92b290b5d97be630a5]) successfully deleted
Sep 26 23:17:53 dist-test-slave-vcv1 krb5kdc[26915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:17:53.337519 26945 catalog_manager.cc:4841] TS 2eb23ce2ec574f2eb01ecfa2b132a8e9 (127.16.62.65:33549): tablet fc08d6d66a474c0d9d2bc855b7852ca3 (table test-table [id=8179fdf6bbbf4a92b290b5d97be630a5]) successfully deleted
I20260926 23:17:53.344259 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.311404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.16.62.65:33549)
0926 23:17:53.311571 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.311577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.316979 (+ 5402us) server_negotiation.cc:407] Connection header received
0926 23:17:53.318566 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.318572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.318678 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.318765 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:53.321616 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.322900 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.332003 (+ 9103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.332791 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.332905 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.338490 (+ 5585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:53.338500 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:53.338513 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:53.338557 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:53.341489 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.342295 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.342301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.342304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.342428 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.342807 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.342813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.342815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.343129 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:53.343332 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.343674 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.343848 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:17:53.344944 27144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:53.441910 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 27021
I20260926 23:17:53.460210 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 27155
Sep 26 23:17:53 dist-test-slave-vcv1 krb5kdc[26915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:17:53.475358 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.460942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.16.62.66:36577)
0926 23:17:53.461103 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.461107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.461124 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:17:53.461291 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.461294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.461356 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.461435 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:53.462557 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.463221 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.464583 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.465173 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.465307 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.469329 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:53.469343 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:53.469361 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:53.469399 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:53.472754 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.473348 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.473356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.473360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.473488 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.473940 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.473945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.473948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.474358 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:53.474609 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.474828 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.475018 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:17:53.476063 27278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:53.564576 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 27155
I20260926 23:17:53.581830 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 27288
Sep 26 23:17:53 dist-test-slave-vcv1 krb5kdc[26915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:17:53.596328 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:17:53.582630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.16.62.67:34815)
0926 23:17:53.582807 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:17:53.582813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:17:53.582873 (+ 60us) server_negotiation.cc:407] Connection header received
0926 23:17:53.583068 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:17:53.583080 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:17:53.583142 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:17:53.583259 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:17:53.584535 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.585268 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.586482 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:17:53.586867 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:17:53.586995 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:17:53.591130 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:17:53.591142 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:17:53.591158 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:17:53.591190 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:17:53.593747 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.594365 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.594372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.594374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.594496 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:17:53.594849 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:17:53.594855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:17:53.594858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:17:53.595182 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:17:53.595475 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 23:17:53.595786 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 23:17:53.595998 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:17:53.596928 27411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:53.684453 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 27288
I20260926 23:17:53.700708 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 26928
I20260926 23:17:53.701807 26990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:17:53.812100 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 26928
2026-09-26T23:17:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4088 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:17:53 dist-test-slave-vcv1 krb5kdc[27467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:17:53 dist-test-slave-vcv1 krb5kdc[27467](info): set up 2 sockets
Sep 26 23:17:53 dist-test-slave-vcv1 krb5kdc[27467](info): commencing operation
krb5kdc: starting...
W20260926 23:17:54.909865 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:17:55 dist-test-slave-vcv1 krb5kdc[27467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:17:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:17:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.62.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.62.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/rangeradmin_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/rangeradmin_127.16.62.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.62.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.62.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/rangerlookup_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/rangerlookup_127.16.62.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.62.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.62.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/HTTP_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/HTTP_127.16.62.84@KRBTEST.COM.keytab.
I20260926 23:17:55.208606 16633 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZEqmPv/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/postgres -l logfile start
2026-09-26 23:17:57.655 UTC [27497] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 23:17:57.655 UTC [27497] LOG: listening on IPv4 address "127.16.62.84", port 37085
2026-09-26 23:17:57.659 UTC [27497] LOG: listening on Unix socket "/tmp/.s.PGSQL.37085"
2026-09-26 23:17:57.681 UTC [27500] LOG: database system was shut down at 2026-09-26 23:17:56 UTC
2026-09-26 23:17:57.684 UTC [27497] LOG: database system is ready to accept connections
I20260926 23:17:58.657882 16633 mini_postgres.cc:96] Postgres bound to 37085
2026-09-26 23:17:58.675 UTC [27508] FATAL: database "slave" does not exist
127.16.62.84:37085 - accepting connections
I20260926 23:17:58.676016 16633 mini_ranger.cc:161] Starting Ranger...
I20260926 23:17:58.699816 16633 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 23:17:59.187049 16633 mini_ranger.cc:87] Created ranger Postgres database
I20260926 23:17:59.187211 16633 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-admin
2026-09-26 23:17:59,480 [I] DB FLAVOR :POSTGRES
2026-09-26 23:17:59,481 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:17:59,481 [I] Checking connection..
2026-09-26 23:17:59,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 23:17:59,715 [I] Checking connection passed.
2026-09-26 23:17:59,716 [I] --------- Verifying version history table ---------
2026-09-26 23:17:59,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:17:59,951 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:17:59,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:18:00,207 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:18:00,208 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 23:18:00,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 23:18:00.459 UTC [27572] WARNING: there is no transaction in progress
2026-09-26 23:18:00,465 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 23:18:00,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:18:00,698 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 23:18:00,699 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:18:00,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 23:18:00,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 23:18:01,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vcv1.c.gcp-upstream.internal','N') ;"
2026-09-26 23:18:01,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:18:01,688 [I] Table x_portal_user does not exist in database ranger
2026-09-26 23:18:01,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:18:01,945 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 23:18:01,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:18:02,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:18:02,449 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 23:18:02,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 23:18:03.428 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.443 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.459 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.475 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.494 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.514 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.583 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.590 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.606 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.617 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.627 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.643 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.653 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.660 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.667 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:03.674 UTC [27680] WARNING: there is no transaction in progress
2026-09-26 23:18:04,293 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 23:18:04,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 23:18:04,539 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 23:18:04,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:18:04,793 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 23:18:04,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:18:05,050 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 23:18:05,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:18:05,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:18:05,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vcv1.c.gcp-upstream.internal';"
2026-09-26 23:18:05,739 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 23:18:05,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 23:18:05,976 [I] DB_PATCHES have already been applied
I20260926 23:18:05.982228 16633 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZEqmPv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 23:18:05.982339 16633 mini_ranger.cc:193] Using host: 127.16.62.84
I20260926 23:18:05.997252 16633 mini_ranger.cc:227] Ranger admin URL: http://127.16.62.84:51727
Sep 26, 2026 11:18:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:18:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:18:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:18:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:18:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:18:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:18:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 23:18:07.106556 16633 mini_ranger.cc:160] Time spent starting Ranger: real 8.431s user 0.000s sys 0.039s
Sep 26, 2026 11:18:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:18:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 11:18:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:18:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:18:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 23:18:28.968057 16633 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.62.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.62.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/rangerkms_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/rangerkms_127.16.62.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.62.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.62.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.62.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/HTTP_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.62.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/HTTP_127.16.62.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 23:18:29 dist-test-slave-vcv1 krb5kdc[27467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464709, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 23:18:29.125712 16633 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 23:18:29.145587 16633 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 23:18:29.981644 16633 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 23:18:29.981771 16633 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-kms
I20260926 23:18:29.981913 16633 mini_ranger_kms.cc:227] Using postgres at 127.16.62.84:37085
2026-09-26 23:18:30,452 [I] DB FLAVOR :POSTGRES
2026-09-26 23:18:30,453 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:18:30,453 [I] Checking connection
2026-09-26 23:18:30,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 23:18:30,686 [I] connection success
2026-09-26 23:18:30,686 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 23:18:30,687 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 23:18:30,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 23:18:30,912 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 23:18:30,912 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:18:30,913 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 23:18:30,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:37085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 23:18:31,172 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 23:18:31.633070 16633 mini_ranger_kms.cc:310] Created kms service
I20260926 23:18:31.844457 16633 mini_ranger_kms.cc:326] Created kudu user
I20260926 23:18:31.889295 16633 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 23:18:31 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464709, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:18:32.288748 16633 mini_ranger_kms.cc:384] Added ranger policy
I20260926 23:18:32.289057 16633 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 23:18:32.289113 16633 mini_ranger_kms.cc:240] Using host: 127.16.62.84
I20260926 23:18:32.310650 16633 mini_ranger_kms.cc:276] Ranger KMS PID: 27860
I20260926 23:18:32.310855 16633 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.62.84:36881
Sep 26, 2026 11:18:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:18:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:18:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:18:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:18:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:18:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 23:18:34 dist-test-slave-vcv1 krb5kdc[27467](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790464714, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.62.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:18:34.686510 16633 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.561s user 0.112s sys 0.380s
I20260926 23:18:34.686781 16633 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 23:18:34.687004 16633 mini_ranger_kms.cc:401] 127.16.62.84:36881/kms/v1/keys
Sep 26 23:18:34 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464714, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.62.84@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
Sep 26 23:18:35 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464714, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.62.84@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
Sep 26 23:18:37 dist-test-slave-vcv1 krb5kdc[27467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:18:37.547767 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:33933
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:33933
--ranger_config_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:36881/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:18:37.910755 27911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:18:37.911082 27911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:18:37.911266 27911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:18:37.918365 27911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:18:37.918499 27911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:18:37.918623 27911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:18:37.918723 27911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:18:37.918815 27911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:18:37.928766 27911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.62.84:36881/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:33933
--ranger_config_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:33933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:18:37.930069 27911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:18:37.931308 27911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:18:37.945369 27916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:18:37.946076 27919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:18:37.946233 27917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:18:37.947691 27911 server_base.cc:1034] running on GCE node
I20260926 23:18:37.948688 27911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:18:37.950311 27911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:18:37.951613 27911 hybrid_clock.cc:648] HybridClock initialized: now 1790464717951502 us; error 39 us; skew 500 ppm
I20260926 23:18:37.952209 27911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:18:37 dist-test-slave-vcv1 krb5kdc[27467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464717, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:18:37.961985 27911 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:18:37.964496 27911 webserver.cc:460] Webserver started at http://127.16.62.126:44389/ using document root <none> and password file <none>
I20260926 23:18:37.965376 27911 fs_manager.cc:359] Metadata directory not provided
I20260926 23:18:37.965564 27911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:18:37.965960 27911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:18:37 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464717, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:18:38.033558 27911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "53b177f3c304451a906cd56f62e21d65"
format_stamp: "Formatted at 2026-09-26 23:18:38 on dist-test-slave-vcv1"
server_key: "eeeea6ff6c923384a756d6157ac4a176"
server_key_iv: "9da549f43479039cec31f99385fb9c66"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:38.034535 27911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "53b177f3c304451a906cd56f62e21d65"
format_stamp: "Formatted at 2026-09-26 23:18:38 on dist-test-slave-vcv1"
server_key: "eeeea6ff6c923384a756d6157ac4a176"
server_key_iv: "9da549f43479039cec31f99385fb9c66"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:38.040958 27911 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:18:38.060079 27926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:18:38.061682 27911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:18:38.061894 27911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "53b177f3c304451a906cd56f62e21d65"
format_stamp: "Formatted at 2026-09-26 23:18:38 on dist-test-slave-vcv1"
server_key: "eeeea6ff6c923384a756d6157ac4a176"
server_key_iv: "9da549f43479039cec31f99385fb9c66"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:38.062080 27911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:18:38.079180 27911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:18:38.083006 27911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:18:38.083328 27911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:18:38.114434 27911 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:33933
I20260926 23:18:38.114796 27977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:33933 every 8 connection(s)
I20260926 23:18:38.115918 27911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:18:38.123235 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 27911
I20260926 23:18:38.123476 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:18:38.123791 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:18:38.124679 16633 external_mini_cluster.cc:1428] Setting key c4c48cd546b819ae8d7cfc3f50ee8b5c
I20260926 23:18:38.146641 27978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65: Bootstrap starting.
I20260926 23:18:38.150079 27978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65: Neither blocks nor log segments found. Creating new log.
I20260926 23:18:38.151108 27978 log.cc:824] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65: Log is configured to *not* fsync() on all Append() calls
I20260926 23:18:38.154596 27978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65: No bootstrap required, opened a new log
Sep 26 23:18:38 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:18:38.165248 27978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b177f3c304451a906cd56f62e21d65" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33933 } }
I20260926 23:18:38.165656 27978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:18:38.165735 27981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:38.127193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.16.62.126:33933)
0926 23:18:38.145177 (+ 17984us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:38.145189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:38.145248 (+ 59us) server_negotiation.cc:407] Connection header received
0926 23:18:38.147324 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:38.147345 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:38.147730 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:38.148121 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:18:38.149510 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:38.150858 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:38.152192 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:38.152862 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:38.153112 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:38.157467 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:18:38.157482 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:18:38.157499 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:18:38.157546 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:18:38.161436 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:38.162102 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:38.162114 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:38.162121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:38.162297 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:38.162699 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:38.162704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:38.162707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:38.163416 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:18:38.163721 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:38.164034 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 23:18:38.164456 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17675,"thread_start_us":17290,"threads_started":1}
I20260926 23:18:38.165788 27978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b177f3c304451a906cd56f62e21d65, State: Initialized, Role: FOLLOWER
I20260926 23:18:38.166319 27978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b177f3c304451a906cd56f62e21d65" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33933 } }
I20260926 23:18:38.166563 27978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:18:38.166702 27978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:18:38.166886 27978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:18:38.169723 27978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b177f3c304451a906cd56f62e21d65" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33933 } }
I20260926 23:18:38.170210 27978 leader_election.cc:302] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53b177f3c304451a906cd56f62e21d65; no voters:
I20260926 23:18:38.170886 27978 leader_election.cc:288] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:18:38.171020 27983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:18:38.171854 27983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [term 1 LEADER]: Becoming Leader. State: Replica: 53b177f3c304451a906cd56f62e21d65, State: Running, Role: LEADER
I20260926 23:18:38.172434 27983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b177f3c304451a906cd56f62e21d65" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33933 } }
I20260926 23:18:38.173171 27978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:18:38.174916 27984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53b177f3c304451a906cd56f62e21d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b177f3c304451a906cd56f62e21d65" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33933 } } }
I20260926 23:18:38.175038 27985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53b177f3c304451a906cd56f62e21d65. Latest consensus state: current_term: 1 leader_uuid: "53b177f3c304451a906cd56f62e21d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b177f3c304451a906cd56f62e21d65" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33933 } } }
I20260926 23:18:38.175434 27985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [sys.catalog]: This master's current role is: LEADER
I20260926 23:18:38.175328 27984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65 [sys.catalog]: This master's current role is: LEADER
I20260926 23:18:38.177168 27988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:18:38.182433 27988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:18:38.183025 27978 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 23:18:38.206774 27988 catalog_manager.cc:1347] Generated new cluster ID: 534c6881fbf647f79406a1cd21fe8e8e
I20260926 23:18:38.206944 27988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:18:38.242141 27988 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:18:38.244232 27988 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:18:38.255923 27988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65: Generated new TSK 0
I20260926 23:18:38.257050 27988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:18:39.034087 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:36881/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:33933
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:18:39 dist-test-slave-vcv1 krb5kdc[27467](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790464719, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 23:18:39.532249 28023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:18:39.532727 28023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:18:39.533028 28023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:18:39.546207 28023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:18:39.546464 28023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:18:39.546746 28023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:18:39.561192 28023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.62.84:36881/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:33933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:18:39.562968 28023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:18:39.564976 28023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:18:39.590793 28033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:18:39.595767 28034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:18:39.598755 28023 server_base.cc:1034] running on GCE node
W20260926 23:18:39.599282 28036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:18:39.600190 28023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:18:39.601678 28023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:18:39.603091 28023 hybrid_clock.cc:648] HybridClock initialized: now 1790464719602882 us; error 90 us; skew 500 ppm
I20260926 23:18:39.603633 28023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:18:39 dist-test-slave-vcv1 krb5kdc[27467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464719, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:18:39.608889 28023 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:18:39.618739 28023 webserver.cc:460] Webserver started at http://127.16.62.65:32863/ using document root <none> and password file <none>
I20260926 23:18:39.619742 28023 fs_manager.cc:359] Metadata directory not provided
I20260926 23:18:39.620028 28023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:18:39.620633 28023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:18:39 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464719, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:18:39.649698 28023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "ede5c795d91a42f6a03f121d51cc9227"
format_stamp: "Formatted at 2026-09-26 23:18:39 on dist-test-slave-vcv1"
server_key: "92735f59e302a24b72b5208de588ca08"
server_key_iv: "192b99b05c121975916b2a370e06aa13"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:39.651345 28023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "ede5c795d91a42f6a03f121d51cc9227"
format_stamp: "Formatted at 2026-09-26 23:18:39 on dist-test-slave-vcv1"
server_key: "92735f59e302a24b72b5208de588ca08"
server_key_iv: "192b99b05c121975916b2a370e06aa13"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:39.670404 28023 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.009s sys 0.008s
I20260926 23:18:39.694460 28043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:18:39.701967 28023 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.000s
I20260926 23:18:39.702199 28023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "ede5c795d91a42f6a03f121d51cc9227"
format_stamp: "Formatted at 2026-09-26 23:18:39 on dist-test-slave-vcv1"
server_key: "92735f59e302a24b72b5208de588ca08"
server_key_iv: "192b99b05c121975916b2a370e06aa13"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:39.702589 28023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:18:39.746027 28023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:18:39.751526 28023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:18:39.751873 28023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:18:39.753430 28023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:18:39.755929 28023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:18:39.756100 28023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:18:39.756383 28023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:18:39.756529 28023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:18:39.944571 28023 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:40583
I20260926 23:18:39.944622 28155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:40583 every 8 connection(s)
I20260926 23:18:39.946321 28023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:18:39.954892 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 28023
I20260926 23:18:39.955121 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:18:39.956054 16633 external_mini_cluster.cc:1428] Setting key b8597573c9288861589f0aa7cfa2e022
Sep 26 23:18:39 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464719, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
Sep 26 23:18:39 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464719, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:18:40.007133 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:39.959449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:33933 (local address 127.16.62.65:60225)
0926 23:18:39.960432 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:39.960479 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:18:39.963122 (+ 2643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:18:39.985504 (+ 22382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:18:39.985522 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:18:39.986258 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:18:39.987117 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:39.987145 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:39.988707 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:39.988713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:39.991254 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:39.991271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:39.993303 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:39.993310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:39.993622 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:39.994445 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:18:39.994476 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:18:39.997954 (+ 3478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:18:40.001974 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:18:40.001984 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:18:40.002005 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:18:40.002593 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:18:40.003036 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:18:40.003064 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:18:40.003068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:18:40.003315 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:18:40.004066 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:18:40.004074 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:18:40.004557 (+ 483us) client_negotiation.cc:769] Sending connection context
0926 23:18:40.004907 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 23:18:40.005521 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":1075,"threads_started":1}
I20260926 23:18:40.009215 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:39.960112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:60225 (local address 127.16.62.126:33933)
0926 23:18:39.984641 (+ 24529us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:39.984649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:39.984674 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:18:39.984763 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:39.984771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:39.984869 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:39.985981 (+ 1112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:18:39.987343 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:39.988505 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:39.991464 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:39.992449 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:39.992573 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:39.998255 (+ 5682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:18:39.998262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:18:39.998267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:18:39.998320 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:18:40.001803 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:40.002757 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:40.002762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:40.002766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:40.002911 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:40.003441 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:40.003446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:40.003450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:40.003853 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:18:40.003974 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:40.007654 (+ 3680us) server_negotiation.cc:299] Negotiation successful
0926 23:18:40.007889 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24374,"thread_start_us":24225,"threads_started":1}
I20260926 23:18:40.010830 28156 heartbeater.cc:347] Connected to a master server at 127.16.62.126:33933
I20260926 23:18:40.011169 28156 heartbeater.cc:464] Registering TS with master...
I20260926 23:18:40.012117 28156 heartbeater.cc:511] Master 127.16.62.126:33933 requested a full tablet report, sending...
I20260926 23:18:40.014590 27943 ts_manager.cc:194] Registered new tserver with Master: ede5c795d91a42f6a03f121d51cc9227 (127.16.62.65:40583)
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
I20260926 23:18:40.020073 27943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:60225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:18:40.061740 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:36881/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:33933
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:18:40 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464719, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
W20260926 23:18:40.559593 28162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:18:40.559974 28162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:18:40.560220 28162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:18:40.571879 28162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:18:40.572094 28162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:18:40.572366 28162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:18:40.589962 28162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.62.84:36881/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:33933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:18:40.591722 28162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:18:40.593448 28162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:18:40.610209 28169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:18:40.621501 28170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:18:40.621898 28172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:18:40.622099 28162 server_base.cc:1034] running on GCE node
I20260926 23:18:40.623199 28162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:18:40.625383 28162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:18:40.626655 28162 hybrid_clock.cc:648] HybridClock initialized: now 1790464720626540 us; error 105 us; skew 500 ppm
I20260926 23:18:40.627209 28162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:18:40 dist-test-slave-vcv1 krb5kdc[27467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:18:40.631560 28162 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:18:40.634804 28162 webserver.cc:460] Webserver started at http://127.16.62.66:43177/ using document root <none> and password file <none>
I20260926 23:18:40.635542 28162 fs_manager.cc:359] Metadata directory not provided
I20260926 23:18:40.635701 28162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:18:40.636046 28162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:18:40 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:18:40.657122 28162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "0db56c8b800b47cfaf24884b7f573240"
format_stamp: "Formatted at 2026-09-26 23:18:40 on dist-test-slave-vcv1"
server_key: "25c72b17d52886b77d6ad3573af9f67c"
server_key_iv: "ac6ea6f3f0a1aae6eba6a13170b66f3b"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:40.658212 28162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "0db56c8b800b47cfaf24884b7f573240"
format_stamp: "Formatted at 2026-09-26 23:18:40 on dist-test-slave-vcv1"
server_key: "25c72b17d52886b77d6ad3573af9f67c"
server_key_iv: "ac6ea6f3f0a1aae6eba6a13170b66f3b"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:40.665383 28162 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260926 23:18:40.682683 28186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:18:40.684536 28162 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:18:40.684798 28162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "0db56c8b800b47cfaf24884b7f573240"
format_stamp: "Formatted at 2026-09-26 23:18:40 on dist-test-slave-vcv1"
server_key: "25c72b17d52886b77d6ad3573af9f67c"
server_key_iv: "ac6ea6f3f0a1aae6eba6a13170b66f3b"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:40.685132 28162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:18:40.711293 28162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:18:40.717321 28162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:18:40.717733 28162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:18:40.718976 28162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:18:40.720769 28162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:18:40.720902 28162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 23:18:40.721069 28162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:18:40.721169 28162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:18:40.775898 28162 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:36543
I20260926 23:18:40.776070 28298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:36543 every 8 connection(s)
I20260926 23:18:40.778121 28162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:18:40.786391 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 28162
I20260926 23:18:40.786592 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:18:40.787210 16633 external_mini_cluster.cc:1428] Setting key 0fed013dff02ac9d5740f97d10d3dc56
Sep 26 23:18:40 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
I20260926 23:18:40.817446 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:40.782668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:45129 (local address 127.16.62.126:33933)
0926 23:18:40.784686 (+ 2018us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:40.784693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:40.796238 (+ 11545us) server_negotiation.cc:407] Connection header received
0926 23:18:40.796334 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:40.796341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:40.796446 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:40.796553 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:18:40.798891 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:40.800213 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:40.803003 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:40.803737 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:40.803865 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:40.809038 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:18:40.809048 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:18:40.809052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:18:40.809100 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:18:40.813541 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:40.814562 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:40.814571 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:40.814575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:40.814718 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:40.815270 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:40.815275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:40.815278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:40.815725 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:18:40.815894 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:40.816774 (+ 880us) server_negotiation.cc:299] Negotiation successful
0926 23:18:40.816995 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"thread_start_us":1750,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
I20260926 23:18:40.818789 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:40.782011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:33933 (local address 127.16.62.66:45129)
0926 23:18:40.789747 (+ 7736us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:40.789804 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 23:18:40.792871 (+ 3067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:18:40.797037 (+ 4166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:18:40.797052 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:18:40.797724 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:18:40.798633 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:40.798660 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:40.800460 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:40.800468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:40.802740 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:40.802763 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:40.804183 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:40.804192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:40.804503 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:40.805343 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:18:40.805377 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:18:40.808774 (+ 3397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:18:40.813742 (+ 4968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:18:40.813754 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:18:40.813774 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:18:40.814398 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:18:40.814869 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:18:40.814877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:18:40.814882 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:18:40.815123 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:18:40.815982 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:18:40.815990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:18:40.816368 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 23:18:40.816679 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 23:18:40.817242 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7436,"thread_start_us":7212,"threads_started":1}
I20260926 23:18:40.820734 28299 heartbeater.cc:347] Connected to a master server at 127.16.62.126:33933
I20260926 23:18:40.821039 28299 heartbeater.cc:464] Registering TS with master...
I20260926 23:18:40.821913 28299 heartbeater.cc:511] Master 127.16.62.126:33933 requested a full tablet report, sending...
I20260926 23:18:40.823585 27943 ts_manager.cc:194] Registered new tserver with Master: 0db56c8b800b47cfaf24884b7f573240 (127.16.62.66:36543)
I20260926 23:18:40.824815 27943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:45129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:18:40.848619 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:36881/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:33933
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:18:41.029440 28156 heartbeater.cc:503] Master 127.16.62.126:33933 was elected leader, sending a full tablet report...
W20260926 23:18:41.222680 28305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:18:41.222946 28305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:18:41.223106 28305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:18:41.230901 28305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:18:41.231016 28305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:18:41.231190 28305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:18:41.241406 28305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.62.84:36881/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:33933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:18:41.242774 28305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:18:41.244081 28305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:18:41.257975 28310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:18:41.258090 28311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:18:41.258126 28313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:18:41.258980 28305 server_base.cc:1034] running on GCE node
I20260926 23:18:41.259744 28305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:18:41.260824 28305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:18:41.262243 28305 hybrid_clock.cc:648] HybridClock initialized: now 1790464721262054 us; error 58 us; skew 500 ppm
I20260926 23:18:41.262634 28305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:18:41 dist-test-slave-vcv1 krb5kdc[27467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:18:41.266563 28305 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:18:41.268278 28305 webserver.cc:460] Webserver started at http://127.16.62.67:44889/ using document root <none> and password file <none>
I20260926 23:18:41.268985 28305 fs_manager.cc:359] Metadata directory not provided
I20260926 23:18:41.269150 28305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:18:41.269479 28305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:18:41 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:18:41.285579 28305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "d1039dee91c74b9c83a8f1c14a65cc7c"
format_stamp: "Formatted at 2026-09-26 23:18:41 on dist-test-slave-vcv1"
server_key: "078aebda54d4d75bf3c49e2cc931bfd7"
server_key_iv: "04a7b389ac8ed540698368ca2ec43061"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:41.286485 28305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "d1039dee91c74b9c83a8f1c14a65cc7c"
format_stamp: "Formatted at 2026-09-26 23:18:41 on dist-test-slave-vcv1"
server_key: "078aebda54d4d75bf3c49e2cc931bfd7"
server_key_iv: "04a7b389ac8ed540698368ca2ec43061"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:41.292682 28305 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 23:18:41.307289 28320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:18:41.308840 28305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:18:41.309082 28305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "d1039dee91c74b9c83a8f1c14a65cc7c"
format_stamp: "Formatted at 2026-09-26 23:18:41 on dist-test-slave-vcv1"
server_key: "078aebda54d4d75bf3c49e2cc931bfd7"
server_key_iv: "04a7b389ac8ed540698368ca2ec43061"
server_key_version: "kuduclusterkey@0"
I20260926 23:18:41.309279 28305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:18:41.323616 28305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:18:41.327570 28305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:18:41.327865 28305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:18:41.328769 28305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:18:41.330070 28305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:18:41.330139 28305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:18:41.330302 28305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:18:41.330354 28305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:18:41.365115 28305 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:39643
I20260926 23:18:41.365157 28432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:39643 every 8 connection(s)
I20260926 23:18:41.366581 28305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:18:41.371390 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 28305
I20260926 23:18:41.371567 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:18:41.372277 16633 external_mini_cluster.cc:1428] Setting key 2da0c1f07efefd71d9eeb406e31b95fd
Sep 26 23:18:41 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:18:41.388070 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.369720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:34455 (local address 127.16.62.126:33933)
0926 23:18:41.370219 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.370225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.371281 (+ 1056us) server_negotiation.cc:407] Connection header received
0926 23:18:41.373638 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.373644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.373733 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.373821 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:18:41.375647 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.376457 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.377911 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.378426 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.378530 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.381827 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:18:41.381832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:18:41.381835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:18:41.381864 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:18:41.384923 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.385629 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.385636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.385640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.385764 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.386232 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.386236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.386238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.386563 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:18:41.386672 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.387504 (+ 832us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.387757 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":211,"threads_started":1}
I20260926 23:18:41.388933 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.370373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:33933 (local address 127.16.62.67:34455)
0926 23:18:41.371089 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.371146 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 23:18:41.373329 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:18:41.374093 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:18:41.374104 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:18:41.374691 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:18:41.375482 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.375514 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.376579 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.376583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.377769 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.377779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.378612 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.378616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.378807 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.379493 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:18:41.379513 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:18:41.381619 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:18:41.385065 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:18:41.385074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:18:41.385091 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:18:41.385504 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:18:41.385880 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:18:41.385886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:18:41.385890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:18:41.386119 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:18:41.386686 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:18:41.386694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:18:41.387001 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 23:18:41.387334 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 23:18:41.387849 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":273,"threads_started":1}
I20260926 23:18:41.390360 28433 heartbeater.cc:347] Connected to a master server at 127.16.62.126:33933
I20260926 23:18:41.390651 28433 heartbeater.cc:464] Registering TS with master...
I20260926 23:18:41.391341 28433 heartbeater.cc:511] Master 127.16.62.126:33933 requested a full tablet report, sending...
I20260926 23:18:41.392789 27943 ts_manager.cc:194] Registered new tserver with Master: d1039dee91c74b9c83a8f1c14a65cc7c (127.16.62.67:39643)
I20260926 23:18:41.393623 27943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:34455
I20260926 23:18:41.399724 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:18:41.417244 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.404265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.16.62.126:33933)
0926 23:18:41.404437 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.404442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.404786 (+ 344us) server_negotiation.cc:407] Connection header received
0926 23:18:41.404934 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.404937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.405004 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.405085 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:18:41.406546 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.407284 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.408701 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.409109 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.409220 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.411846 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:18:41.411853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:18:41.411856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:18:41.411889 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:18:41.415057 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.415627 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.415630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.415632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.415756 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.416078 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.416081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.416083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.416387 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:18:41.416556 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.416848 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.416999 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:18:41.421360 27943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:18:41.428508 27943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:18:41.455838 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.441399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.16.62.67:39643)
0926 23:18:41.442025 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.442035 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.443230 (+ 1195us) server_negotiation.cc:407] Connection header received
0926 23:18:41.443523 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.443530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.443791 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.443991 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:18:41.445232 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.447346 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.451056 (+ 3710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.453422 (+ 2366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.453563 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.453882 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.455129 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.455303 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":517,"threads_started":1}
I20260926 23:18:41.456807 28448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.441045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:39643 (local address 127.0.0.1:54534)
0926 23:18:41.443059 (+ 2014us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.443107 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:18:41.443287 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:18:41.444370 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:18:41.444381 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:18:41.444422 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:18:41.444972 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.444988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.447589 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.447597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.450820 (+ 3223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.450841 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.454123 (+ 3282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.454131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.454436 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.454488 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 23:18:41.456081 (+ 1593us) client_negotiation.cc:240] Negotiation successful
0926 23:18:41.456387 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793,"thread_start_us":1648,"threads_started":1}
I20260926 23:18:41.460359 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.442282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40583 (local address 127.0.0.1:51218)
0926 23:18:41.449520 (+ 7238us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.449545 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:18:41.449796 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:18:41.450905 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:18:41.450912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:18:41.450945 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:18:41.451454 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.451467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.455284 (+ 3817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.455288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.457391 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.457403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.459500 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.459506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.459690 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.459733 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:18:41.459863 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 23:18:41.459990 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7071,"thread_start_us":1552,"threads_started":1}
I20260926 23:18:41.460412 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.441798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.16.62.65:40583)
0926 23:18:41.442529 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.442540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.449671 (+ 7131us) server_negotiation.cc:407] Connection header received
0926 23:18:41.450008 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.450020 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.450353 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.450546 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:18:41.451697 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.453769 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.457568 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.459041 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.459136 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.459334 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.459917 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.460079 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":415,"threads_started":1}
I20260926 23:18:41.460585 28368 tablet_service.cc:1467] Processing CreateTablet for tablet 79d9d57f4d2d4788b4d11e77638d6def (DEFAULT_TABLE table=test-table [id=b3294e3790dd4e7c85bb72d0ce7db2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:18:41.461970 28368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d9d57f4d2d4788b4d11e77638d6def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:18:41.464013 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.443793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:36543 (local address 127.0.0.1:36238)
0926 23:18:41.456658 (+ 12865us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.456677 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:18:41.456803 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:18:41.457413 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:18:41.457417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:18:41.457441 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:18:41.457889 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.457901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.460649 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.460666 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.461978 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.461989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.463253 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.463264 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.463422 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.463453 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:18:41.463561 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 23:18:41.463724 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12749,"thread_start_us":12597,"threads_started":1}
I20260926 23:18:41.465456 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.444052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.16.62.66:36543)
0926 23:18:41.455262 (+ 11210us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.455271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.456798 (+ 1527us) server_negotiation.cc:407] Connection header received
0926 23:18:41.456939 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.456949 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.457228 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.457427 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:18:41.458063 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.459679 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.462175 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.463038 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.463140 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.463261 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.463613 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.463766 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11045,"thread_start_us":10931,"threads_started":1}
I20260926 23:18:41.466655 28091 tablet_service.cc:1467] Processing CreateTablet for tablet 79d9d57f4d2d4788b4d11e77638d6def (DEFAULT_TABLE table=test-table [id=b3294e3790dd4e7c85bb72d0ce7db2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:18:41.468122 28091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d9d57f4d2d4788b4d11e77638d6def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:18:41.468982 28234 tablet_service.cc:1467] Processing CreateTablet for tablet 79d9d57f4d2d4788b4d11e77638d6def (DEFAULT_TABLE table=test-table [id=b3294e3790dd4e7c85bb72d0ce7db2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:18:41.470247 28234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d9d57f4d2d4788b4d11e77638d6def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:18:41.476362 28452 tablet_bootstrap.cc:492] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: Bootstrap starting.
I20260926 23:18:41.480284 28452 tablet_bootstrap.cc:654] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: Neither blocks nor log segments found. Creating new log.
I20260926 23:18:41.480818 28453 tablet_bootstrap.cc:492] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: Bootstrap starting.
I20260926 23:18:41.482052 28452 log.cc:824] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:18:41.484511 28455 tablet_bootstrap.cc:492] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: Bootstrap starting.
I20260926 23:18:41.484607 28453 tablet_bootstrap.cc:654] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: Neither blocks nor log segments found. Creating new log.
I20260926 23:18:41.485679 28453 log.cc:824] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: Log is configured to *not* fsync() on all Append() calls
I20260926 23:18:41.485971 28452 tablet_bootstrap.cc:492] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: No bootstrap required, opened a new log
I20260926 23:18:41.486249 28452 ts_tablet_manager.cc:1397] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 23:18:41.488129 28455 tablet_bootstrap.cc:654] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: Neither blocks nor log segments found. Creating new log.
I20260926 23:18:41.489728 28455 log.cc:824] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: Log is configured to *not* fsync() on all Append() calls
I20260926 23:18:41.490549 28453 tablet_bootstrap.cc:492] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: No bootstrap required, opened a new log
I20260926 23:18:41.490829 28453 ts_tablet_manager.cc:1397] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260926 23:18:41.492766 28455 tablet_bootstrap.cc:492] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: No bootstrap required, opened a new log
I20260926 23:18:41.493008 28455 ts_tablet_manager.cc:1397] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 23:18:41.498062 28452 raft_consensus.cc:348] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } }
I20260926 23:18:41.498508 28452 raft_consensus.cc:374] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:18:41.498605 28452 raft_consensus.cc:729] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1039dee91c74b9c83a8f1c14a65cc7c, State: Initialized, Role: FOLLOWER
I20260926 23:18:41.498823 28453 raft_consensus.cc:348] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } }
I20260926 23:18:41.499261 28453 raft_consensus.cc:374] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:18:41.499384 28453 raft_consensus.cc:729] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede5c795d91a42f6a03f121d51cc9227, State: Initialized, Role: FOLLOWER
I20260926 23:18:41.499259 28452 consensus_queue.cc:260] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } }
I20260926 23:18:41.499948 28453 consensus_queue.cc:260] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } }
I20260926 23:18:41.501850 28453 ts_tablet_manager.cc:1428] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 23:18:41.504951 28455 raft_consensus.cc:348] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } }
I20260926 23:18:41.505446 28455 raft_consensus.cc:374] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:18:41.505537 28455 raft_consensus.cc:729] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db56c8b800b47cfaf24884b7f573240, State: Initialized, Role: FOLLOWER
I20260926 23:18:41.505823 28433 heartbeater.cc:503] Master 127.16.62.126:33933 was elected leader, sending a full tablet report...
I20260926 23:18:41.506748 28452 ts_tablet_manager.cc:1428] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: Time spent starting tablet: real 0.020s user 0.007s sys 0.012s
I20260926 23:18:41.506210 28455 consensus_queue.cc:260] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } }
I20260926 23:18:41.507740 28299 heartbeater.cc:503] Master 127.16.62.126:33933 was elected leader, sending a full tablet report...
I20260926 23:18:41.508442 28455 ts_tablet_manager.cc:1428] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260926 23:18:41.515413 28460 raft_consensus.cc:482] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:18:41.515695 28460 raft_consensus.cc:504] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } }
I20260926 23:18:41.517349 28460 leader_election.cc:288] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1039dee91c74b9c83a8f1c14a65cc7c (127.16.62.67:39643), ede5c795d91a42f6a03f121d51cc9227 (127.16.62.65:40583)
I20260926 23:18:41.528443 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.517721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:35573 (local address 127.16.62.67:39643)
0926 23:18:41.517915 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.517921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.518959 (+ 1038us) server_negotiation.cc:407] Connection header received
0926 23:18:41.519274 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.519277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.519364 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.519428 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:18:41.520494 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.521902 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.525640 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.527023 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.527201 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.527300 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.527966 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.528097 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:18:41.529232 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.517983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:56305 (local address 127.16.62.65:40583)
0926 23:18:41.518153 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.518158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.519596 (+ 1438us) server_negotiation.cc:407] Connection header received
0926 23:18:41.519811 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.519844 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.519951 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.520308 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:18:41.521080 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.522459 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.526115 (+ 3656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.527650 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.527818 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.527906 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.528785 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.528922 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 23:18:41.529388 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.518274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40583 (local address 127.16.62.66:56305)
0926 23:18:41.519486 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.519510 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:18:41.519633 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:18:41.520442 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:18:41.520446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:18:41.520468 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:18:41.520875 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.520900 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.522676 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.522681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.525808 (+ 3127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.525821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.527883 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.527889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.528054 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.528084 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:18:41.528755 (+ 671us) client_negotiation.cc:240] Negotiation successful
0926 23:18:41.528946 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":107,"thread_start_us":728,"threads_started":1}
I20260926 23:18:41.529474 28461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.517692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:39643 (local address 127.16.62.66:35573)
0926 23:18:41.518820 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.518852 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 23:18:41.519097 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:18:41.519685 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:18:41.519691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:18:41.519722 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:18:41.520277 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.520293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.522056 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.522062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.525474 (+ 3412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:18:41.525488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.527531 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.527537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:18:41.527768 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.527813 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 23:18:41.528751 (+ 938us) client_negotiation.cc:240] Negotiation successful
0926 23:18:41.528915 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":26,"thread_start_us":534,"threads_started":1}
I20260926 23:18:41.530465 28388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d9d57f4d2d4788b4d11e77638d6def" candidate_uuid: "0db56c8b800b47cfaf24884b7f573240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" is_pre_election: true
I20260926 23:18:41.530791 28111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d9d57f4d2d4788b4d11e77638d6def" candidate_uuid: "0db56c8b800b47cfaf24884b7f573240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede5c795d91a42f6a03f121d51cc9227" is_pre_election: true
I20260926 23:18:41.531056 28388 raft_consensus.cc:2454] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db56c8b800b47cfaf24884b7f573240 in term 0.
I20260926 23:18:41.531562 28111 raft_consensus.cc:2454] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db56c8b800b47cfaf24884b7f573240 in term 0.
I20260926 23:18:41.531762 28189 leader_election.cc:302] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db56c8b800b47cfaf24884b7f573240, d1039dee91c74b9c83a8f1c14a65cc7c; no voters:
I20260926 23:18:41.532186 28460 raft_consensus.cc:2781] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:18:41.532341 28460 raft_consensus.cc:482] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:18:41.532462 28460 raft_consensus.cc:3037] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:18:41.536616 28460 raft_consensus.cc:504] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } }
I20260926 23:18:41.537436 28460 leader_election.cc:288] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [CANDIDATE]: Term 1 election: Requested vote from peers d1039dee91c74b9c83a8f1c14a65cc7c (127.16.62.67:39643), ede5c795d91a42f6a03f121d51cc9227 (127.16.62.65:40583)
I20260926 23:18:41.537942 28388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d9d57f4d2d4788b4d11e77638d6def" candidate_uuid: "0db56c8b800b47cfaf24884b7f573240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c"
I20260926 23:18:41.537950 28111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d9d57f4d2d4788b4d11e77638d6def" candidate_uuid: "0db56c8b800b47cfaf24884b7f573240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede5c795d91a42f6a03f121d51cc9227"
I20260926 23:18:41.538169 28388 raft_consensus.cc:3037] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:18:41.538177 28111 raft_consensus.cc:3037] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [term 0 FOLLOWER]: Advancing to term 1
W20260926 23:18:41.539578 28300 tablet.cc:2369] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:18:41.541550 28111 raft_consensus.cc:2454] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db56c8b800b47cfaf24884b7f573240 in term 1.
I20260926 23:18:41.541520 28388 raft_consensus.cc:2454] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db56c8b800b47cfaf24884b7f573240 in term 1.
I20260926 23:18:41.541970 28189 leader_election.cc:302] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db56c8b800b47cfaf24884b7f573240, d1039dee91c74b9c83a8f1c14a65cc7c; no voters:
I20260926 23:18:41.542418 28460 raft_consensus.cc:2781] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:18:41.542901 28460 raft_consensus.cc:686] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 1 LEADER]: Becoming Leader. State: Replica: 0db56c8b800b47cfaf24884b7f573240, State: Running, Role: LEADER
I20260926 23:18:41.543660 28460 consensus_queue.cc:237] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } }
I20260926 23:18:41.549918 27942 catalog_manager.cc:5495] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0db56c8b800b47cfaf24884b7f573240 (127.16.62.66). New cstate: current_term: 1 leader_uuid: "0db56c8b800b47cfaf24884b7f573240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0db56c8b800b47cfaf24884b7f573240" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 36543 } health_report { overall_health: HEALTHY } } }
I20260926 23:18:41.585660 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.580283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.16.62.66:36543)
0926 23:18:41.580507 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.580512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.580533 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:18:41.580664 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.580668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.580743 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.580897 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:18:41.581286 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.581988 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.583428 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.583894 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.584038 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.584599 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:18:41.585093 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:18:41.585248 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.585307 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.585382 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:18:41.593815 28388 raft_consensus.cc:1261] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [term 1 FOLLOWER]: Refusing update from remote peer 0db56c8b800b47cfaf24884b7f573240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:18:41.593847 28111 raft_consensus.cc:1261] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [term 1 FOLLOWER]: Refusing update from remote peer 0db56c8b800b47cfaf24884b7f573240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:18:41.594774 28463 consensus_queue.cc:1035] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede5c795d91a42f6a03f121d51cc9227" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:18:41.595103 28460 consensus_queue.cc:1035] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1039dee91c74b9c83a8f1c14a65cc7c" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:18:41.622236 27942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:18:41.622509 27942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:18:41.625670 27942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53b177f3c304451a906cd56f62e21d65: Sending DeleteTablet for 3 replicas of tablet 79d9d57f4d2d4788b4d11e77638d6def
I20260926 23:18:41.626643 28368 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d9d57f4d2d4788b4d11e77638d6def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:18:41 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:54534
I20260926 23:18:41.626785 28091 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d9d57f4d2d4788b4d11e77638d6def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:18:41 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:51218
I20260926 23:18:41.628216 28234 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d9d57f4d2d4788b4d11e77638d6def with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:18:41 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:36238
I20260926 23:18:41.628396 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 28023
I20260926 23:18:41.630064 28476 tablet_replica.cc:330] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: stopping tablet replica
I20260926 23:18:41.630455 28476 raft_consensus.cc:2229] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:18:41.630924 28476 raft_consensus.cc:2258] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:18:41.631556 28477 tablet_replica.cc:330] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: stopping tablet replica
I20260926 23:18:41.631927 28477 raft_consensus.cc:2229] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:18:41.632368 28477 raft_consensus.cc:2258] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:18:41.633272 28478 tablet_replica.cc:330] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: stopping tablet replica
I20260926 23:18:41.633641 28478 raft_consensus.cc:2229] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:18:41.634240 28478 raft_consensus.cc:2258] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:18:41.634963 28476 ts_tablet_manager.cc:1905] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:18:41.636209 28478 ts_tablet_manager.cc:1905] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:18:41.641855 28476 ts_tablet_manager.cc:1918] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:18:41.642010 28476 log.cc:1196] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/79d9d57f4d2d4788b4d11e77638d6def
I20260926 23:18:41.642521 28476 ts_tablet_manager.cc:1939] T 79d9d57f4d2d4788b4d11e77638d6def P d1039dee91c74b9c83a8f1c14a65cc7c: Deleting consensus metadata
I20260926 23:18:41.643491 28478 ts_tablet_manager.cc:1918] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:18:41.643667 28478 log.cc:1196] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/79d9d57f4d2d4788b4d11e77638d6def
I20260926 23:18:41.644191 28478 ts_tablet_manager.cc:1939] T 79d9d57f4d2d4788b4d11e77638d6def P 0db56c8b800b47cfaf24884b7f573240: Deleting consensus metadata
I20260926 23:18:41.644374 27929 catalog_manager.cc:4841] TS d1039dee91c74b9c83a8f1c14a65cc7c (127.16.62.67:39643): tablet 79d9d57f4d2d4788b4d11e77638d6def (table test-table [id=b3294e3790dd4e7c85bb72d0ce7db2e0]) successfully deleted
Sep 26 23:18:41 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:18:41.645412 28477 ts_tablet_manager.cc:1905] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:18:41.645778 27929 catalog_manager.cc:4841] TS 0db56c8b800b47cfaf24884b7f573240 (127.16.62.66:36543): tablet 79d9d57f4d2d4788b4d11e77638d6def (table test-table [id=b3294e3790dd4e7c85bb72d0ce7db2e0]) successfully deleted
I20260926 23:18:41.652631 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.629644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.16.62.65:40583)
0926 23:18:41.629825 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.629832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.633554 (+ 3722us) server_negotiation.cc:407] Connection header received
0926 23:18:41.633885 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.633890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.633976 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.634052 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:18:41.636119 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.637171 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.639207 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.639845 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.639950 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.646417 (+ 6467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:18:41.646429 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:18:41.646445 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:18:41.646480 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:18:41.649616 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.650312 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.650318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.650321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.650454 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.650835 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.650840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.650850 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.651208 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:18:41.651515 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.651809 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.652300 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:18:41.653218 28477 ts_tablet_manager.cc:1918] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:18:41.653354 28151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:18:41.653375 28477 log.cc:1196] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/79d9d57f4d2d4788b4d11e77638d6def
I20260926 23:18:41.744874 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 28023
I20260926 23:18:41.744952 28477 ts_tablet_manager.cc:1939] T 79d9d57f4d2d4788b4d11e77638d6def P ede5c795d91a42f6a03f121d51cc9227: Deleting consensus metadata
W20260926 23:18:41.763691 27930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.62.65:40583 (error 108)
I20260926 23:18:41.764688 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 28162
I20260926 23:18:41.765702 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.765135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40583 (local address 127.0.0.1:51240)
0926 23:18:41.765389 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.765512 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 23:18:41.765937 27930 catalog_manager.cc:4584] TS ede5c795d91a42f6a03f121d51cc9227 (127.16.62.65:40583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79d9d57f4d2d4788b4d11e77638d6def: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40583: connect: Connection refused (error 111)
Sep 26 23:18:41 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:18:41.782567 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.765834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.16.62.66:36543)
0926 23:18:41.766015 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.766021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.766040 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:18:41.766101 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.766105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.766203 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.766301 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:18:41.767592 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.768500 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.769810 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.770251 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.770385 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.774983 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:18:41.774998 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:18:41.775016 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:18:41.775081 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:18:41.779444 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.780292 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.780302 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.780306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.780485 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.780918 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.780923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.780926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.781358 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:18:41.781692 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.781979 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.782205 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:18:41.783342 28294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:18:41.819355 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.818901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40583 (local address 127.0.0.1:51246)
0926 23:18:41.819104 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.819204 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:18:41.820335 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.819958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40583 (local address 127.0.0.1:51248)
0926 23:18:41.820092 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.820216 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 23:18:41.881014 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.880524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40583 (local address 127.0.0.1:51254)
0926 23:18:41.880768 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.880853 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 23:18:41.881968 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.881643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40583 (local address 127.0.0.1:51262)
0926 23:18:41.881783 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.881855 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 23:18:41.897801 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 28162
I20260926 23:18:41.912441 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 28305
Sep 26 23:18:41 dist-test-slave-vcv1 krb5kdc[27467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:18:41.928889 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.913244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.16.62.67:39643)
0926 23:18:41.913497 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:18:41.913505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:18:41.913624 (+ 119us) server_negotiation.cc:407] Connection header received
0926 23:18:41.913888 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:18:41.913894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:18:41.913976 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:18:41.914130 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:18:41.915635 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.916541 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.917932 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:18:41.918360 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:18:41.918500 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:18:41.923215 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:18:41.923228 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:18:41.923243 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:18:41.923275 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:18:41.926182 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.926835 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.926842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.926845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.926973 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:18:41.927399 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:18:41.927403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:18:41.927405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:18:41.927802 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:18:41.928072 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 23:18:41.928443 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 23:18:41.928619 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 23:18:41.929585 28428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:18:41.962318 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.961853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40583 (local address 127.0.0.1:51264)
0926 23:18:41.962052 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.962165 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 23:18:41.963390 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:18:41.963009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40583 (local address 127.0.0.1:51272)
0926 23:18:41.963198 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 23:18:41.963276 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 23:18:42.019824 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 28305
I20260926 23:18:42.036098 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 27911
I20260926 23:18:42.037336 27973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:18:42.154197 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 27911
I20260926 23:18:42.179697 16633 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 23:18:42.516086 16633 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 23:18:42.516359 16633 mini_ranger.cc:66] Stopping Ranger...
I20260926 23:18:42.861406 16633 mini_ranger.cc:68] Stopped Ranger
2026-09-26 23:18:42.861 UTC [27497] LOG: received smart shutdown request
2026-09-26 23:18:42.865 UTC [27497] LOG: background worker "logical replication launcher" (PID 27506) exited with exit code 1
2026-09-26 23:18:42.865 UTC [27501] LOG: shutting down
2026-09-26 23:18:43.040 UTC [27497] LOG: database system is shut down
2026-09-26T23:18:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49352 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:18:43 dist-test-slave-vcv1 krb5kdc[28493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:18:43 dist-test-slave-vcv1 krb5kdc[28493](info): set up 2 sockets
Sep 26 23:18:43 dist-test-slave-vcv1 krb5kdc[28493](info): commencing operation
krb5kdc: starting...
W20260926 23:18:44.305315 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:18:44 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:18:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.62.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.62.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/rangeradmin_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/rangeradmin_127.16.62.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.62.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.62.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/rangerlookup_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/rangerlookup_127.16.62.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.62.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.62.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/HTTP_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/HTTP_127.16.62.84@KRBTEST.COM.keytab.
I20260926 23:18:44.631815 16633 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZEqmPv/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/postgres -l logfile start
2026-09-26 23:18:47.249 UTC [28523] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 23:18:47.249 UTC [28523] LOG: listening on IPv4 address "127.16.62.84", port 48511
2026-09-26 23:18:47.253 UTC [28523] LOG: listening on Unix socket "/tmp/.s.PGSQL.48511"
2026-09-26 23:18:47.278 UTC [28526] LOG: database system was shut down at 2026-09-26 23:18:45 UTC
2026-09-26 23:18:47.281 UTC [28523] LOG: database system is ready to accept connections
I20260926 23:18:48.257640 16633 mini_postgres.cc:96] Postgres bound to 48511
2026-09-26 23:18:48.279 UTC [28534] FATAL: database "slave" does not exist
127.16.62.84:48511 - accepting connections
I20260926 23:18:48.279937 16633 mini_ranger.cc:161] Starting Ranger...
I20260926 23:18:48.306602 16633 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 23:18:48.802471 16633 mini_ranger.cc:87] Created ranger Postgres database
I20260926 23:18:48.802593 16633 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-admin
2026-09-26 23:18:49,091 [I] DB FLAVOR :POSTGRES
2026-09-26 23:18:49,091 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:18:49,091 [I] Checking connection..
2026-09-26 23:18:49,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 23:18:49,344 [I] Checking connection passed.
2026-09-26 23:18:49,344 [I] --------- Verifying version history table ---------
2026-09-26 23:18:49,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:18:49,580 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:18:49,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:18:49,837 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:18:49,837 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 23:18:49,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 23:18:50.065 UTC [28598] WARNING: there is no transaction in progress
2026-09-26 23:18:50,072 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 23:18:50,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:18:50,314 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 23:18:50,314 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:18:50,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 23:18:50,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 23:18:50,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vcv1.c.gcp-upstream.internal','N') ;"
2026-09-26 23:18:51,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:18:51,327 [I] Table x_portal_user does not exist in database ranger
2026-09-26 23:18:51,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:18:51,588 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 23:18:51,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:18:51,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:18:52,097 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 23:18:52,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 23:18:53.088 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.105 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.122 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.138 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.155 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.176 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.241 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.248 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.264 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.275 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.285 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.301 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.312 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.320 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.328 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:53.336 UTC [28706] WARNING: there is no transaction in progress
2026-09-26 23:18:54,013 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 23:18:54,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 23:18:54,255 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 23:18:54,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:18:54,484 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 23:18:54,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:18:54,724 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 23:18:54,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:18:54,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:18:55,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vcv1.c.gcp-upstream.internal';"
2026-09-26 23:18:55,439 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 23:18:55,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 23:18:55,667 [I] DB_PATCHES have already been applied
I20260926 23:18:55.673378 16633 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZEqmPv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 23:18:55.673466 16633 mini_ranger.cc:193] Using host: 127.16.62.84
I20260926 23:18:55.687994 16633 mini_ranger.cc:227] Ranger admin URL: http://127.16.62.84:34481
Sep 26, 2026 11:18:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:18:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:18:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 23:18:56.814780 16633 mini_ranger.cc:160] Time spent starting Ranger: real 8.535s user 0.004s sys 0.037s
Sep 26, 2026 11:19:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:19:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 11:19:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:19:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:19:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 23:19:17.811095 16633 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.62.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.62.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/rangerkms_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.62.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/rangerkms_127.16.62.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.62.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.62.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.62.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/HTTP_127.16.62.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.62.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/HTTP_127.16.62.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 23:19:17 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464757, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 23:19:17.977706 16633 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 23:19:17.998634 16633 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 23:19:18.912828 16633 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 23:19:18.912921 16633 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-kms
I20260926 23:19:18.912955 16633 mini_ranger_kms.cc:227] Using postgres at 127.16.62.84:48511
2026-09-26 23:19:19,267 [I] DB FLAVOR :POSTGRES
2026-09-26 23:19:19,268 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:19:19,268 [I] Checking connection
2026-09-26 23:19:19,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 23:19:19,508 [I] connection success
2026-09-26 23:19:19,509 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 23:19:19,509 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 23:19:19,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 23:19:19,740 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 23:19:19,741 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:19:19,741 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 23:19:19,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.62.84:48511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 23:19:20,016 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 23:19:20.464449 16633 mini_ranger_kms.cc:310] Created kms service
I20260926 23:19:20.665195 16633 mini_ranger_kms.cc:326] Created kudu user
I20260926 23:19:20.712176 16633 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 23:19:20 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464757, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:21.100952 16633 mini_ranger_kms.cc:384] Added ranger policy
I20260926 23:19:21.101284 16633 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZEqmPv/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZEqmPv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 23:19:21.101351 16633 mini_ranger_kms.cc:240] Using host: 127.16.62.84
I20260926 23:19:21.117506 16633 mini_ranger_kms.cc:276] Ranger KMS PID: 28886
I20260926 23:19:21.117640 16633 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.62.84:42893
Sep 26, 2026 11:19:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:19:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:19:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:19:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:19:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:19:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 23:19:23 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790464763, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.62.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:23.456049 16633 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.478s user 0.073s sys 0.300s
I20260926 23:19:23.456316 16633 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 23:19:23.456493 16633 mini_ranger_kms.cc:401] 127.16.62.84:42893/kms/v1/keys
Sep 26 23:19:23 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464763, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.62.84@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
Sep 26 23:19:24 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464763, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.62.84@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
Sep 26 23:19:26 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:19:26.415973 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44375
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:44375
--ranger_config_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:42893/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:26.757030 28937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:26.757391 28937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:26.757512 28937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:26.764861 28937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:19:26.764967 28937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:26.765012 28937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:26.765069 28937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:19:26.765123 28937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:19:26.775010 28937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.62.84:42893/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:44375
--ranger_config_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:26.776234 28937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:26.777487 28937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:26.789542 28942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:26.789629 28943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:26.789652 28945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:26.789971 28937 server_base.cc:1034] running on GCE node
I20260926 23:19:26.790961 28937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:26.792452 28937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:26.793689 28937 hybrid_clock.cc:648] HybridClock initialized: now 1790464766793648 us; error 32 us; skew 500 ppm
I20260926 23:19:26.794032 28937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:26 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:26.801651 28937 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:19:26.803184 28937 webserver.cc:460] Webserver started at http://127.16.62.126:43671/ using document root <none> and password file <none>
I20260926 23:19:26.803777 28937 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:26.803870 28937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:26.804206 28937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:19:26 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:26.854815 28937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "7c5169813f4a4ae48e912d6a9c783add"
format_stamp: "Formatted at 2026-09-26 23:19:26 on dist-test-slave-vcv1"
server_key: "1278e9b8dcade6411afbf59a67e2c229"
server_key_iv: "5d26adb835d034f18bf37cd72c44072e"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:26.855643 28937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "7c5169813f4a4ae48e912d6a9c783add"
format_stamp: "Formatted at 2026-09-26 23:19:26 on dist-test-slave-vcv1"
server_key: "1278e9b8dcade6411afbf59a67e2c229"
server_key_iv: "5d26adb835d034f18bf37cd72c44072e"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:26.861462 28937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 23:19:26.878494 28952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:26.880223 28937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 23:19:26.880436 28937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "7c5169813f4a4ae48e912d6a9c783add"
format_stamp: "Formatted at 2026-09-26 23:19:26 on dist-test-slave-vcv1"
server_key: "1278e9b8dcade6411afbf59a67e2c229"
server_key_iv: "5d26adb835d034f18bf37cd72c44072e"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:26.880673 28937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:26.901554 28937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:26.905746 28937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:26.906066 28937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:26.924774 28937 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:44375
I20260926 23:19:26.924808 29003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:44375 every 8 connection(s)
I20260926 23:19:26.926151 28937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:19:26.930079 29004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:26.930861 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 28937
I20260926 23:19:26.931121 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:19:26.932108 16633 external_mini_cluster.cc:1428] Setting key 3852c392f687cc6b30d1dfb04dc8e803
I20260926 23:19:26.943624 29004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Bootstrap starting.
Sep 26 23:19:26 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:26.947161 29004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:26.948549 29004 log.cc:824] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:26.951644 29004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: No bootstrap required, opened a new log
I20260926 23:19:26.954039 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:26.934296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.16.62.126:44375)
0926 23:19:26.935127 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:26.935141 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:26.935182 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:19:26.937074 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:26.937098 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:26.937490 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:26.937884 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:26.939193 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:26.940749 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:26.942067 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:26.942729 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:26.942918 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:26.946846 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:26.946855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:26.946869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:26.946940 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:26.950126 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:26.950791 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:26.950805 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:26.950810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:26.950964 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:26.951351 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:26.951358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:26.951361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:26.951905 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:26.952256 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:26.952679 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 23:19:26.953074 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":416,"threads_started":1}
I20260926 23:19:26.960276 29004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } }
I20260926 23:19:26.960664 29004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:26.960780 29004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5169813f4a4ae48e912d6a9c783add, State: Initialized, Role: FOLLOWER
I20260926 23:19:26.961414 29004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } }
I20260926 23:19:26.961761 29004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:19:26.961943 29004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:19:26.962157 29004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:26.964491 29004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } }
I20260926 23:19:26.965058 29004 leader_election.cc:302] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c5169813f4a4ae48e912d6a9c783add; no voters:
I20260926 23:19:26.965649 29004 leader_election.cc:288] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:19:26.965807 29009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:19:26.966507 29009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 1 LEADER]: Becoming Leader. State: Replica: 7c5169813f4a4ae48e912d6a9c783add, State: Running, Role: LEADER
I20260926 23:19:26.967056 29009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } }
I20260926 23:19:26.967922 29004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:19:26.970165 29010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c5169813f4a4ae48e912d6a9c783add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } } }
I20260926 23:19:26.970176 29011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c5169813f4a4ae48e912d6a9c783add. Latest consensus state: current_term: 1 leader_uuid: "7c5169813f4a4ae48e912d6a9c783add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } } }
I20260926 23:19:26.970613 29010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:26.970654 29011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:26.972074 29015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:19:26.977344 29015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:19:26.978307 29004 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 23:19:26.998795 29015 catalog_manager.cc:1347] Generated new cluster ID: 06d6729e7b774b8589c215a16b42767e
I20260926 23:19:26.999010 29015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:19:27.013581 29015 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:19:27.015091 29015 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:19:27.032244 29015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Generated new TSK 0
I20260926 23:19:27.033243 29015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:19:27.812107 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:42893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44375
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:19:28 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790464768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 23:19:28.231518 29049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:28.231993 29049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:28.232213 29049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:28.244234 29049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:28.244393 29049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:28.244648 29049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:19:28.260927 29049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.62.84:42893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:28.262745 29049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:28.264459 29049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:28.284233 29059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:28.289021 29060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:28.297063 29049 server_base.cc:1034] running on GCE node
W20260926 23:19:28.298149 29062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:28.299055 29049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:28.300556 29049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:28.304157 29049 hybrid_clock.cc:648] HybridClock initialized: now 1790464768304019 us; error 64 us; skew 500 ppm
I20260926 23:19:28.304932 29049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:28 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:28.310561 29049 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:19:28.314584 29049 webserver.cc:460] Webserver started at http://127.16.62.65:41517/ using document root <none> and password file <none>
I20260926 23:19:28.315595 29049 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:28.315783 29049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:28.316214 29049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:19:28 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:28.334969 29049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "1281a4a912e14744a72944b3453e111d"
format_stamp: "Formatted at 2026-09-26 23:19:28 on dist-test-slave-vcv1"
server_key: "a619b5b00378670fdef3fdc3507650f6"
server_key_iv: "59d0cd214ff60fc815223c4f2d39810b"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:28.336117 29049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "1281a4a912e14744a72944b3453e111d"
format_stamp: "Formatted at 2026-09-26 23:19:28 on dist-test-slave-vcv1"
server_key: "a619b5b00378670fdef3fdc3507650f6"
server_key_iv: "59d0cd214ff60fc815223c4f2d39810b"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:28.343881 29049 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260926 23:19:28.376881 29069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:28.379019 29049 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.008s sys 0.004s
I20260926 23:19:28.379293 29049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "1281a4a912e14744a72944b3453e111d"
format_stamp: "Formatted at 2026-09-26 23:19:28 on dist-test-slave-vcv1"
server_key: "a619b5b00378670fdef3fdc3507650f6"
server_key_iv: "59d0cd214ff60fc815223c4f2d39810b"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:28.379629 29049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:28.434748 29049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:28.440368 29049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:28.440776 29049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:28.455513 29049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:28.470508 29049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:28.470736 29049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:19:28.470960 29049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:28.471099 29049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:19:28 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:28.878901 29049 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:36487
I20260926 23:19:28.878986 29181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:36487 every 8 connection(s)
I20260926 23:19:28.880633 29049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:19:28.881063 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 29049
I20260926 23:19:28.881271 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:19:28.882086 16633 external_mini_cluster.cc:1428] Setting key 8c339f9a29524d25f4d9d7e97a5c7adc
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Sep 26 23:19:28 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:28.951912 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:28.898299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:44995 (local address 127.16.62.126:44375)
0926 23:19:28.915994 (+ 17695us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:28.916018 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:28.921421 (+ 5403us) server_negotiation.cc:407] Connection header received
0926 23:19:28.930173 (+ 8752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:28.930180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:28.930302 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:28.930390 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:28.932896 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:28.934060 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:28.937570 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:28.938359 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:28.938487 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:28.943574 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:28.943584 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:28.943590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:28.943650 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:28.947619 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:28.948745 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:28.948753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:28.948760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:28.948921 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:28.949515 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:28.949523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:28.949529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:28.949927 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:28.950064 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:28.951115 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0926 23:19:28.951402 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17486,"thread_start_us":17728,"threads_started":1}
I20260926 23:19:28.954484 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:28.899078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44375 (local address 127.16.62.65:44995)
0926 23:19:28.921188 (+ 22110us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:28.921262 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:28.923893 (+ 2631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:28.931052 (+ 7159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:28.931066 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:28.931734 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:28.932635 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:28.932672 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:28.934256 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:28.934260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:28.937382 (+ 3122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:28.937401 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:28.938900 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:28.938910 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:28.939271 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:28.940196 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:28.940234 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:28.943268 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:28.947848 (+ 4580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:28.947859 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:28.947881 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:28.948594 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:28.949077 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:28.949084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:28.949091 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:28.949382 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:28.950171 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:28.950181 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:28.950665 (+ 484us) client_negotiation.cc:769] Sending connection context
0926 23:19:28.952321 (+ 1656us) client_negotiation.cc:240] Negotiation successful
0926 23:19:28.952936 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21729,"thread_start_us":21448,"threads_started":1}
I20260926 23:19:28.956501 29182 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44375
I20260926 23:19:28.956835 29182 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:28.957741 29182 heartbeater.cc:511] Master 127.16.62.126:44375 requested a full tablet report, sending...
I20260926 23:19:28.960301 28969 ts_manager.cc:194] Registered new tserver with Master: 1281a4a912e14744a72944b3453e111d (127.16.62.65:36487)
I20260926 23:19:28.962671 28969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:44995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:19:28.975268 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:42893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44375
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:19:29 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
W20260926 23:19:29.483225 29188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:29.483489 29188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:29.483670 29188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:29.491166 29188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:29.491325 29188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:29.491493 29188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:19:29.501235 29188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.62.84:42893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:29.502610 29188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:29.503934 29188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:29.517068 29202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:29.517031 29203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:29.517828 29205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:29.518257 29188 server_base.cc:1034] running on GCE node
I20260926 23:19:29.518884 29188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:29.519858 29188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:29.521132 29188 hybrid_clock.cc:648] HybridClock initialized: now 1790464769521044 us; error 52 us; skew 500 ppm
I20260926 23:19:29.521502 29188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:29 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464769, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:29.524729 29188 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:19:29.526391 29188 webserver.cc:460] Webserver started at http://127.16.62.66:37963/ using document root <none> and password file <none>
I20260926 23:19:29.527091 29188 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:29.527238 29188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:29.527559 29188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:19:29 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464769, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:29.546720 29188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "004558a7227c401badb352d7dedd1b44"
format_stamp: "Formatted at 2026-09-26 23:19:29 on dist-test-slave-vcv1"
server_key: "6e307d24952bf3d2e6680809d6a69703"
server_key_iv: "aed8f6c5690453a7334c499c9f416f9c"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:29.547785 29188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "004558a7227c401badb352d7dedd1b44"
format_stamp: "Formatted at 2026-09-26 23:19:29 on dist-test-slave-vcv1"
server_key: "6e307d24952bf3d2e6680809d6a69703"
server_key_iv: "aed8f6c5690453a7334c499c9f416f9c"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:29.555002 29188 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 23:19:29.574572 29212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:29.576617 29188 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:19:29.576798 29188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "004558a7227c401badb352d7dedd1b44"
format_stamp: "Formatted at 2026-09-26 23:19:29 on dist-test-slave-vcv1"
server_key: "6e307d24952bf3d2e6680809d6a69703"
server_key_iv: "aed8f6c5690453a7334c499c9f416f9c"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:29.577077 29188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:29.606762 29188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:29.610739 29188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:29.611038 29188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:29.612059 29188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:29.613458 29188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:29.613533 29188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:29.613677 29188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:29.613739 29188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:29.648597 29188 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:35199
I20260926 23:19:29.648617 29324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:35199 every 8 connection(s)
I20260926 23:19:29.649875 29188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:19:29.657023 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 29188
I20260926 23:19:29.657214 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:19:29.657819 16633 external_mini_cluster.cc:1428] Setting key 441a570ebf01d9f8cc422223fc8cbd29
Sep 26 23:19:29 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464769, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:29.670842 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:29.652660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:55323 (local address 127.16.62.126:44375)
0926 23:19:29.653182 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:29.653186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:29.654129 (+ 943us) server_negotiation.cc:407] Connection header received
0926 23:19:29.655959 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:29.655965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:29.656061 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:29.656205 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:29.657706 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:29.658607 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:29.660700 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:29.661123 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:29.661264 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:29.664788 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:29.664793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:29.664796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:29.664826 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:29.667504 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:29.668333 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:29.668342 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:29.668344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:29.668513 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:29.668963 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:29.668970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:29.668975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:29.669423 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:29.669572 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:29.670250 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 23:19:29.670466 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":267,"threads_started":1}
I20260926 23:19:29.671795 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:29.653235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44375 (local address 127.16.62.66:55323)
0926 23:19:29.653967 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:29.654008 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:29.655688 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:29.656432 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:29.656442 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:29.656850 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:29.657535 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:29.657560 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:29.658752 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:29.658760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:29.660559 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:29.660571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:29.661354 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:29.661362 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:29.661607 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:29.662315 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:29.662337 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:29.664537 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:29.667648 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:29.667658 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:29.667676 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:29.668205 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:29.668634 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:29.668641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:29.668644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:29.668843 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:29.669567 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:29.669575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:29.669880 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 23:19:29.670144 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 23:19:29.670650 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":316,"threads_started":1}
I20260926 23:19:29.673465 29325 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44375
I20260926 23:19:29.673784 29325 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:29.674567 29325 heartbeater.cc:511] Master 127.16.62.126:44375 requested a full tablet report, sending...
I20260926 23:19:29.676265 28969 ts_manager.cc:194] Registered new tserver with Master: 004558a7227c401badb352d7dedd1b44 (127.16.62.66:35199)
I20260926 23:19:29.677371 28969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:55323
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:19:29.719538 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:42893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44375
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:19:29.966702 29182 heartbeater.cc:503] Master 127.16.62.126:44375 was elected leader, sending a full tablet report...
W20260926 23:19:30.045680 29331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:30.046007 29331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:30.046185 29331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:30.053475 29331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:30.053629 29331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:30.053813 29331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:19:30.063467 29331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.62.84:42893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:30.064769 29331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:30.066033 29331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:30.080040 29337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:30.079998 29336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:30.079998 29339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:30.080679 29331 server_base.cc:1034] running on GCE node
I20260926 23:19:30.081337 29331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:30.082262 29331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:30.083470 29331 hybrid_clock.cc:648] HybridClock initialized: now 1790464770083405 us; error 51 us; skew 500 ppm
I20260926 23:19:30.083781 29331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:30 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464770, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:30.087186 29331 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:19:30.088594 29331 webserver.cc:460] Webserver started at http://127.16.62.67:36597/ using document root <none> and password file <none>
I20260926 23:19:30.089252 29331 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:30.089381 29331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:30.089687 29331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:19:30 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464770, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:30.102764 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "63076fa3f11a4c4880890b7fb68de1b2"
format_stamp: "Formatted at 2026-09-26 23:19:30 on dist-test-slave-vcv1"
server_key: "10065b2eef6c62d252b2202b6ed63785"
server_key_iv: "480cf09115bbfebc6af127b766d62322"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:30.103807 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "63076fa3f11a4c4880890b7fb68de1b2"
format_stamp: "Formatted at 2026-09-26 23:19:30 on dist-test-slave-vcv1"
server_key: "10065b2eef6c62d252b2202b6ed63785"
server_key_iv: "480cf09115bbfebc6af127b766d62322"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:30.109189 29331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 23:19:30.122749 29346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:30.124244 29331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:19:30.124420 29331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "63076fa3f11a4c4880890b7fb68de1b2"
format_stamp: "Formatted at 2026-09-26 23:19:30 on dist-test-slave-vcv1"
server_key: "10065b2eef6c62d252b2202b6ed63785"
server_key_iv: "480cf09115bbfebc6af127b766d62322"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:30.124660 29331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:30.156386 29331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:30.160530 29331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:30.160837 29331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:30.161794 29331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:30.163132 29331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:30.163211 29331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:30.163388 29331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:30.163437 29331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:30.196444 29331 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:44135
I20260926 23:19:30.196491 29458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:44135 every 8 connection(s)
I20260926 23:19:30.197736 29331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:19:30.202250 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 29331
I20260926 23:19:30.202425 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:19:30.203075 16633 external_mini_cluster.cc:1428] Setting key 3a2c7104c54648f878980a0144fc1daf
Sep 26 23:19:30 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464770, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:30.218852 29461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:30.200276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:57359 (local address 127.16.62.126:44375)
0926 23:19:30.200699 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:30.200704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:30.201691 (+ 987us) server_negotiation.cc:407] Connection header received
0926 23:19:30.204264 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:30.204270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:30.204424 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:30.204540 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:30.206407 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.207171 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.208776 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.209210 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.209311 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:30.212733 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:30.212738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:30.212741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:30.212780 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:30.215262 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:30.215990 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:30.215994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:30.215997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:30.216184 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:30.216623 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:30.216627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:30.216630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:30.217012 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:30.217181 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:30.218149 (+ 968us) server_negotiation.cc:299] Negotiation successful
0926 23:19:30.218383 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":213,"threads_started":1}
I20260926 23:19:30.219705 29462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:30.200837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44375 (local address 127.16.62.67:57359)
0926 23:19:30.201529 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:30.201570 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:30.203907 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:30.204813 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:30.204821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:30.205399 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:30.206220 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:30.206247 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.207299 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.207302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:30.208649 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:30.208660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.209436 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.209441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:30.209670 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:30.210348 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:30.210371 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:30.212510 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:30.215399 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:30.215405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:30.215417 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:30.215850 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:30.216293 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:30.216296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:30.216299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:30.216481 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:30.217262 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:30.217269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:30.217668 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 23:19:30.217985 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 23:19:30.218496 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":291,"threads_started":1}
I20260926 23:19:30.221107 29459 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44375
I20260926 23:19:30.221396 29459 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:30.222070 29459 heartbeater.cc:511] Master 127.16.62.126:44375 requested a full tablet report, sending...
I20260926 23:19:30.223423 28969 ts_manager.cc:194] Registered new tserver with Master: 63076fa3f11a4c4880890b7fb68de1b2 (127.16.62.67:44135)
I20260926 23:19:30.224296 28969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:57359
I20260926 23:19:30.231371 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:19:30.233217 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 29049
Sep 26 23:19:30 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:19:30.250380 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:30.234274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.16.62.65:36487)
0926 23:19:30.234885 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:30.234891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:30.234916 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:19:30.234970 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:30.234976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:30.235170 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:30.235299 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:30.237013 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.238062 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.239544 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.240126 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.240319 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:30.244620 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:30.244630 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:30.244644 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:30.244679 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:30.247546 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:30.248281 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:30.248290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:30.248292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:30.248424 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:30.248807 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:30.248816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:30.248819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:30.249142 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:30.249469 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:30.249770 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 23:19:30.250028 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":330,"threads_started":1}
I20260926 23:19:30.251684 29177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:30.337947 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 29049
I20260926 23:19:30.353773 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 29188
Sep 26 23:19:30 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:19:30.369957 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:30.354675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.16.62.66:35199)
0926 23:19:30.355179 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:30.355186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:30.355213 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:19:30.355283 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:30.355289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:30.355499 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:30.355664 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:30.357142 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.358131 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.359588 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.360183 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.360343 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:30.364485 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:30.364495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:30.364509 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:30.364547 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:30.367034 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:30.367744 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:30.367753 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:30.367756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:30.367890 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:30.368408 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:30.368413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:30.368416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:30.368755 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:30.368978 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:30.369323 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 23:19:30.369590 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260926 23:19:30.371212 29320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:30.453852 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 29188
I20260926 23:19:30.470307 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 29331
Sep 26 23:19:30 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:19:30.487197 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:30.471507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.16.62.67:44135)
0926 23:19:30.472186 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:30.472194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:30.472222 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:19:30.472279 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:30.472285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:30.472488 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:30.472614 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:30.473997 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.475113 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.476561 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:30.477081 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:30.477182 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:30.481479 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:30.481489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:30.481509 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:30.481544 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:30.484460 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:30.485146 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:30.485153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:30.485155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:30.485275 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:30.485672 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:30.485678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:30.485681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:30.485983 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:30.486306 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:30.486594 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 23:19:30.486842 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":328,"threads_started":1}
I20260926 23:19:30.488494 29454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:30.575861 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 29331
I20260926 23:19:30.591549 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 28937
I20260926 23:19:30.592787 28999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:30.707314 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 28937
I20260926 23:19:30.733292 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44375
--webserver_interface=127.16.62.126
--webserver_port=43671
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:44375
--ranger_config_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:42893/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.62.84:42893/kms with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:31.058825 29471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:31.059136 29471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:31.059227 29471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:31.066453 29471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:19:31.066560 29471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:31.066656 29471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:31.066766 29471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:19:31.066854 29471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:19:31.076946 29471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.62.84:42893/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:44375
--ranger_config_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=43671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:31.078269 29471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:31.079597 29471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:31.091037 29479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:31.091007 29477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:31.091305 29476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:31.091918 29471 server_base.cc:1034] running on GCE node
I20260926 23:19:31.092773 29471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:31.094098 29471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:31.095295 29471 hybrid_clock.cc:648] HybridClock initialized: now 1790464771095252 us; error 42 us; skew 500 ppm
I20260926 23:19:31.095654 29471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:31 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:31.098738 29471 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:19:31.100096 29471 webserver.cc:460] Webserver started at http://127.16.62.126:43671/ using document root <none> and password file <none>
I20260926 23:19:31.100778 29471 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:31.100914 29471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:19:31 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:31.177603 29471 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:19:31.181417 29487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:31.182977 29471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260926 23:19:31.183175 29471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "7c5169813f4a4ae48e912d6a9c783add"
format_stamp: "Formatted at 2026-09-26 23:19:26 on dist-test-slave-vcv1"
server_key: "1278e9b8dcade6411afbf59a67e2c229"
server_key_iv: "5d26adb835d034f18bf37cd72c44072e"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:31.184136 29471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:31.215137 29471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:31.219018 29471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:31.219321 29471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:31.238252 29471 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:44375
I20260926 23:19:31.238270 29538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:44375 every 8 connection(s)
I20260926 23:19:31.239776 29471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:19:31.246737 29539 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:19:31.249706 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 29471
I20260926 23:19:31.250052 29539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Bootstrap starting.
I20260926 23:19:31.250804 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:42893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:36487
--local_ip_for_outbound_sockets=127.16.62.65
--tserver_master_addrs=127.16.62.126:44375
--webserver_port=41517
--webserver_interface=127.16.62.65
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.62.84:42893/kms with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:19:31.269742 29539 log.cc:824] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:31.278684 29539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:19:31.279215 29539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Bootstrap complete.
I20260926 23:19:31.289270 29539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } }
I20260926 23:19:31.290114 29539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5169813f4a4ae48e912d6a9c783add, State: Initialized, Role: FOLLOWER
I20260926 23:19:31.290702 29539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } }
I20260926 23:19:31.290938 29539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:19:31.291064 29539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:19:31.291245 29539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:19:31.294899 29539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } }
I20260926 23:19:31.295394 29539 leader_election.cc:302] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c5169813f4a4ae48e912d6a9c783add; no voters:
I20260926 23:19:31.296044 29539 leader_election.cc:288] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:19:31.296211 29543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:19:31.297535 29543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [term 2 LEADER]: Becoming Leader. State: Replica: 7c5169813f4a4ae48e912d6a9c783add, State: Running, Role: LEADER
I20260926 23:19:31.298115 29543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } }
I20260926 23:19:31.298352 29539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:19:31.300869 29544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7c5169813f4a4ae48e912d6a9c783add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } } }
I20260926 23:19:31.301328 29544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:31.306159 29545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c5169813f4a4ae48e912d6a9c783add. Latest consensus state: current_term: 2 leader_uuid: "7c5169813f4a4ae48e912d6a9c783add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5169813f4a4ae48e912d6a9c783add" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44375 } } }
I20260926 23:19:31.307111 29545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:31.308637 29549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:19:31.314149 29549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:19:31.318297 29549 catalog_manager.cc:1259] Loaded cluster ID: 06d6729e7b774b8589c215a16b42767e
I20260926 23:19:31.318449 29549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:19:31.322237 29549 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:19:31.323659 29549 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Loaded TSK: 0
I20260926 23:19:31.325205 29549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:19:31.328640 29539 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 23:19:31.695164 29541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:31.695586 29541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:31.695780 29541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:31.704684 29541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:31.704869 29541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:31.705089 29541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:19:31.719825 29541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.62.84:42893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:36487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=41517
--tserver_master_addrs=127.16.62.126:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:31.721789 29541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:31.723251 29541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:31.782094 29576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:31.786358 29577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:31.802500 29579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:31.802551 29541 server_base.cc:1034] running on GCE node
I20260926 23:19:31.803462 29541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:31.804843 29541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:31.806180 29541 hybrid_clock.cc:648] HybridClock initialized: now 1790464771806107 us; error 55 us; skew 500 ppm
I20260926 23:19:31.806538 29541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:31 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:31.817612 29541 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:19:31.837685 29541 webserver.cc:460] Webserver started at http://127.16.62.65:41517/ using document root <none> and password file <none>
I20260926 23:19:31.838677 29541 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:31.838930 29541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:19:31 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:31.888981 29541 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.007s sys 0.002s
I20260926 23:19:31.898940 29587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:31.900802 29541 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.001s
I20260926 23:19:31.900995 29541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "1281a4a912e14744a72944b3453e111d"
format_stamp: "Formatted at 2026-09-26 23:19:28 on dist-test-slave-vcv1"
server_key: "a619b5b00378670fdef3fdc3507650f6"
server_key_iv: "59d0cd214ff60fc815223c4f2d39810b"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:31.902093 29541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:31.915777 29541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:31.919920 29541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:31.920262 29541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:31.921443 29541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:31.923017 29541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:31.923185 29541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:19:31.923362 29541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:31.923509 29541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:31.997931 29541 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:36487
I20260926 23:19:31.998068 29700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:36487 every 8 connection(s)
I20260926 23:19:31.999758 29541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:19:32.004681 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 29541
I20260926 23:19:32.005784 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:42893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:35199
--local_ip_for_outbound_sockets=127.16.62.66
--tserver_master_addrs=127.16.62.126:44375
--webserver_port=37963
--webserver_interface=127.16.62.66
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.62.84:42893/kms with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:19:32 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:32.056771 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:32.003022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:39119 (local address 127.16.62.126:44375)
0926 23:19:32.021381 (+ 18359us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:32.021407 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:32.021471 (+ 64us) server_negotiation.cc:407] Connection header received
0926 23:19:32.022234 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:32.022256 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:32.022840 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:32.023298 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:32.025288 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:32.027366 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:32.030134 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:32.031288 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:32.031499 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:32.045363 (+ 13864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:32.045375 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:32.045399 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:32.045461 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:32.050956 (+ 5495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:32.052202 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:32.052247 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:32.052256 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:32.052487 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:32.053072 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:32.053077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:32.053081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:32.053746 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:32.053970 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:32.054993 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0926 23:19:32.055394 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18055,"thread_start_us":18382,"threads_started":1}
I20260926 23:19:32.056789 29704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:32.003211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44375 (local address 127.16.62.65:39119)
0926 23:19:32.018397 (+ 15186us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:32.018462 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:32.021462 (+ 3000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:32.023499 (+ 2037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:32.023513 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:32.024220 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:32.025017 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:32.025044 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:32.027578 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:32.027586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:32.029943 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:32.029963 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:32.031562 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:32.031571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:32.031920 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:32.032853 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:32.035841 (+ 2988us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:32.044940 (+ 9099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:32.051165 (+ 6225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:32.051178 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:32.051197 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:32.051950 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:32.052640 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:32.052646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:32.052650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:32.052937 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:32.053913 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:32.053922 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:32.054375 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 23:19:32.054741 (+ 366us) client_negotiation.cc:240] Negotiation successful
0926 23:19:32.055325 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14821,"thread_start_us":14594,"threads_started":1}
I20260926 23:19:32.060602 29701 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44375
I20260926 23:19:32.060935 29701 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:32.061793 29701 heartbeater.cc:511] Master 127.16.62.126:44375 requested a full tablet report, sending...
W20260926 23:19:32.065261 29701 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:44375 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:19:32.065492 29701 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:32.065771 29701 heartbeater.cc:511] Master 127.16.62.126:44375 requested a full tablet report, sending...
I20260926 23:19:32.066946 29701 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:32.067207 29701 heartbeater.cc:511] Master 127.16.62.126:44375 requested a full tablet report, sending...
W20260926 23:19:32.068312 29701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:19:32.407809 29705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:32.408252 29705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:32.408432 29705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:32.418339 29705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:32.418440 29705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:32.418613 29705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:19:32.431905 29705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.62.84:42893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:35199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=37963
--tserver_master_addrs=127.16.62.126:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:32.433641 29705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:32.435038 29705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:32.458761 29719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:32.458782 29720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:32.460422 29722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:32.465868 29705 server_base.cc:1034] running on GCE node
I20260926 23:19:32.466625 29705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:32.467705 29705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:32.468945 29705 hybrid_clock.cc:648] HybridClock initialized: now 1790464772468880 us; error 57 us; skew 500 ppm
I20260926 23:19:32.469349 29705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:32 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464772, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:32.473803 29705 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:19:32.477098 29705 webserver.cc:460] Webserver started at http://127.16.62.66:37963/ using document root <none> and password file <none>
I20260926 23:19:32.478062 29705 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:32.478225 29705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:19:32 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464772, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:32.521549 29705 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:19:32.531698 29732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:32.533742 29705 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.006s
I20260926 23:19:32.534051 29705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "004558a7227c401badb352d7dedd1b44"
format_stamp: "Formatted at 2026-09-26 23:19:29 on dist-test-slave-vcv1"
server_key: "6e307d24952bf3d2e6680809d6a69703"
server_key_iv: "aed8f6c5690453a7334c499c9f416f9c"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:32.535625 29705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:32.567999 29705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:32.572311 29705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:32.572599 29705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:32.573704 29705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:32.575237 29705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:32.575358 29705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:32.575501 29705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:32.575588 29705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:19:32 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790464772, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:32.644973 29705 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:35199
I20260926 23:19:32.645792 29844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:35199 every 8 connection(s)
I20260926 23:19:32.649308 29705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:19:32.652663 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 29705
I20260926 23:19:32.653486 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.62.84:42893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:44135
--local_ip_for_outbound_sockets=127.16.62.67
--tserver_master_addrs=127.16.62.126:44375
--webserver_port=36597
--webserver_interface=127.16.62.67
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.62.84:42893/kms with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:19:32 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464772, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:32.700990 29847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:32.656463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:54211 (local address 127.16.62.126:44375)
0926 23:19:32.676651 (+ 20188us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:32.676658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:32.676681 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:19:32.676750 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:32.676756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:32.676842 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:32.676974 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:32.679567 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:32.680808 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:32.684130 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:32.684860 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:32.685003 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:32.691326 (+ 6323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:32.691337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:32.691342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:32.691531 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:32.695339 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:32.696352 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:32.696358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:32.696362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:32.696541 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:32.697097 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:32.697102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:32.697105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:32.697465 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:32.697619 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:32.700373 (+ 2754us) server_negotiation.cc:299] Negotiation successful
0926 23:19:32.700606 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20031,"thread_start_us":20884,"threads_started":1}
I20260926 23:19:32.702301 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:32.657049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44375 (local address 127.16.62.66:54211)
0926 23:19:32.672839 (+ 15790us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:32.672913 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:32.675848 (+ 2935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:32.677548 (+ 1700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:32.677565 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:32.678394 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:32.679329 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:32.679359 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:32.680989 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:32.680998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:32.683918 (+ 2920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:32.683938 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:32.685132 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:32.685143 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:32.685511 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:32.686846 (+ 1335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:32.686903 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:32.690810 (+ 3907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:32.695510 (+ 4700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:32.695520 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:32.695540 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:32.696217 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:32.696675 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:32.696680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:32.696684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:32.696953 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:32.697614 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:32.697622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:32.698528 (+ 906us) client_negotiation.cc:769] Sending connection context
0926 23:19:32.699059 (+ 531us) client_negotiation.cc:240] Negotiation successful
0926 23:19:32.699769 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15404,"thread_start_us":15173,"threads_started":1}
I20260926 23:19:32.704674 29845 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44375
I20260926 23:19:32.705034 29845 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:32.706182 29845 heartbeater.cc:511] Master 127.16.62.126:44375 requested a full tablet report, sending...
I20260926 23:19:32.708596 29504 ts_manager.cc:194] Registered new tserver with Master: 004558a7227c401badb352d7dedd1b44 (127.16.62.66:35199)
I20260926 23:19:32.710726 29504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:54211
Sep 26 23:19:33 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464772, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:33.068475 29701 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:33.068909 29701 heartbeater.cc:511] Master 127.16.62.126:44375 requested a full tablet report, sending...
I20260926 23:19:33.070575 29504 ts_manager.cc:194] Registered new tserver with Master: 1281a4a912e14744a72944b3453e111d (127.16.62.65:36487)
I20260926 23:19:33.071774 29504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:39119
W20260926 23:19:33.130661 29849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:33.131167 29849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:33.131361 29849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:33.142640 29849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:33.142892 29849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:33.143126 29849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:19:33.158500 29849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.62.84:42893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:44135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=36597
--tserver_master_addrs=127.16.62.126:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:33.160116 29849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:33.161836 29849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:33.177407 29855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:33.178560 29856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:33.186448 29858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:33.187120 29849 server_base.cc:1034] running on GCE node
I20260926 23:19:33.187882 29849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:33.189296 29849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:33.190589 29849 hybrid_clock.cc:648] HybridClock initialized: now 1790464773190530 us; error 44 us; skew 500 ppm
I20260926 23:19:33.190974 29849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:33 dist-test-slave-vcv1 krb5kdc[28493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464773, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:33.195183 29849 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:19:33.197120 29849 webserver.cc:460] Webserver started at http://127.16.62.67:36597/ using document root <none> and password file <none>
I20260926 23:19:33.198028 29849 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:33.198179 29849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:19:33 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464773, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:33.238337 29849 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.011s sys 0.007s
I20260926 23:19:33.248557 29866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:33.253763 29849 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260926 23:19:33.253954 29849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "63076fa3f11a4c4880890b7fb68de1b2"
format_stamp: "Formatted at 2026-09-26 23:19:30 on dist-test-slave-vcv1"
server_key: "10065b2eef6c62d252b2202b6ed63785"
server_key_iv: "480cf09115bbfebc6af127b766d62322"
server_key_version: "kuduclusterkey@0"
I20260926 23:19:33.255360 29849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:33.321354 29849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:33.326722 29849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:33.327076 29849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:33.333549 29849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:33.345038 29849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:33.345239 29849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:19:33.345391 29849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:33.345494 29849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:19:33 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464772, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for HTTP/127.16.62.84@KRBTEST.COM
I20260926 23:19:33.543140 29849 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:44135
I20260926 23:19:33.543223 29979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:44135 every 8 connection(s)
I20260926 23:19:33.545032 29849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:19:33.549959 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 29849
Sep 26 23:19:33 dist-test-slave-vcv1 krb5kdc[28493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464773, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:33.590608 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.551133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.16.62.126:44375)
0926 23:19:33.569619 (+ 18486us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.569625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.569648 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:19:33.569713 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.569718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.569824 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.569953 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:33.572177 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.573341 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.577166 (+ 3825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.577920 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.578061 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.581546 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:33.581554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:33.581559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:33.581609 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:33.586391 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:33.587449 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:33.587455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:33.587460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:33.587677 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:33.588914 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:33.588919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:33.588923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:33.589275 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:33.589536 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.589968 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.590212 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":18352,"thread_start_us":19054,"threads_started":1}
I20260926 23:19:33.593952 29983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.558861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:44715 (local address 127.16.62.126:44375)
0926 23:19:33.570548 (+ 11687us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.570553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.570573 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:19:33.574082 (+ 3509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.574087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.574177 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.574278 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:33.577167 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.579080 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.581447 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.582187 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.582313 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.587218 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:33.587224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:33.587229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:33.587264 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:33.590693 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:33.591611 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:33.591616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:33.591620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:33.591732 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:33.592184 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:33.592189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:33.592192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:33.592523 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:33.592687 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.593360 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.593555 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":11546,"thread_start_us":10506,"threads_started":1}
I20260926 23:19:33.594918 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.559411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44375 (local address 127.16.62.67:44715)
0926 23:19:33.568868 (+ 9457us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:33.568943 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:33.573815 (+ 4872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:33.574764 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:33.574777 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:33.575457 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:33.576942 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.576969 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.579263 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.579268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.581259 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.581275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.582618 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.582625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.582908 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.584327 (+ 1419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:33.584356 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:33.586944 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:33.590864 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:33.590874 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:33.590892 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:33.591473 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:33.591841 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:33.591844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:33.591846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:33.592025 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:33.592666 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:33.592674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:33.593008 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 23:19:33.593274 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 23:19:33.593815 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9075,"thread_start_us":8793,"threads_started":1}
I20260926 23:19:33.596443 29980 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44375
I20260926 23:19:33.596797 29980 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:33.597611 29980 heartbeater.cc:511] Master 127.16.62.126:44375 requested a full tablet report, sending...
I20260926 23:19:33.599103 29504 ts_manager.cc:194] Registered new tserver with Master: 63076fa3f11a4c4880890b7fb68de1b2 (127.16.62.67:44135)
I20260926 23:19:33.600023 29504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:44715
I20260926 23:19:33.605948 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:19:33.624166 29983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.609483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.16.62.126:44375)
0926 23:19:33.609669 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.609675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.610162 (+ 487us) server_negotiation.cc:407] Connection header received
0926 23:19:33.610293 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.610298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.610370 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.610456 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:33.612036 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.613214 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.614584 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.615228 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.615356 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.618161 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:33.618168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:33.618174 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:33.618214 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:33.621454 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:33.622141 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:33.622146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:33.622150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:33.622289 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:33.622617 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:33.622621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:33.622624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:33.622970 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:33.623216 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.623540 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.623777 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:19:33.629290 29504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:19:33.636947 29504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:19:33.678979 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.653989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:44135 (local address 127.0.0.1:42486)
0926 23:19:33.663932 (+ 9943us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:33.663973 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:33.665325 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:33.666484 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:33.666495 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:33.666545 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:33.667027 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.667044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.670420 (+ 3376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.670429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.674566 (+ 4137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.674594 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.677619 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.677630 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.677934 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.678007 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 23:19:33.678174 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 23:19:33.678441 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9661,"thread_start_us":9494,"threads_started":1}
I20260926 23:19:33.680409 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.653684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.16.62.67:44135)
0926 23:19:33.665825 (+ 12141us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.665834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.665871 (+ 37us) server_negotiation.cc:407] Connection header received
0926 23:19:33.665975 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.665983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.666262 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.666463 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:33.668229 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.670228 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.674785 (+ 4557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.676915 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.677083 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.677392 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.679702 (+ 2310us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.679935 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11960,"thread_start_us":12849,"threads_started":1}
I20260926 23:19:33.686152 29915 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce0b288ab84656b9679b7a7ea321f4 (DEFAULT_TABLE table=test-table [id=5719320eb3244aab9e981c327b0f159f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:19:33.688120 29915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce0b288ab84656b9679b7a7ea321f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:33.692902 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.668335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:35199 (local address 127.0.0.1:47192)
0926 23:19:33.681200 (+ 12865us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:33.681224 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:33.681379 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:33.682589 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:33.682597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:33.682633 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:33.683134 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.683148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.685680 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.685688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.688830 (+ 3142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.688850 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.691751 (+ 2901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.691760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.692009 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.692063 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 23:19:33.692228 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 23:19:33.692416 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12689,"thread_start_us":12733,"threads_started":1}
I20260926 23:19:33.694378 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.664943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.16.62.66:35199)
0926 23:19:33.680422 (+ 15479us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.680434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.681521 (+ 1087us) server_negotiation.cc:407] Connection header received
0926 23:19:33.681649 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.681662 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.682064 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.682308 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:33.683373 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.685465 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.689048 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.691032 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.691209 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.691524 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.693653 (+ 2129us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.693891 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15242,"thread_start_us":15895,"threads_started":1}
I20260926 23:19:33.699362 29780 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce0b288ab84656b9679b7a7ea321f4 (DEFAULT_TABLE table=test-table [id=5719320eb3244aab9e981c327b0f159f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:19:33.700707 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce0b288ab84656b9679b7a7ea321f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:33.706564 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.654193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:36487 (local address 127.0.0.1:56228)
0926 23:19:33.679323 (+ 25130us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:33.679351 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:33.679490 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:33.700416 (+ 20926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:33.700422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:33.700458 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:33.700672 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.700683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.702778 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.702783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.703910 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.703921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.705737 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.705742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.705935 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.705977 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:19:33.706093 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 23:19:33.706272 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24994,"thread_start_us":46137,"threads_started":1}
I20260926 23:19:33.706624 30005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.654503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.16.62.65:36487)
0926 23:19:33.699672 (+ 45169us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.699684 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.699721 (+ 37us) server_negotiation.cc:407] Connection header received
0926 23:19:33.699853 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.699863 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.700161 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.700853 (+ 692us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:33.701106 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.702632 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.704308 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.705571 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.705727 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.705940 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.706163 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.706313 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44934,"thread_start_us":44759,"threads_started":1}
I20260926 23:19:33.708316 30008 tablet_bootstrap.cc:492] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: Bootstrap starting.
I20260926 23:19:33.710525 29636 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce0b288ab84656b9679b7a7ea321f4 (DEFAULT_TABLE table=test-table [id=5719320eb3244aab9e981c327b0f159f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:19:33.711877 30008 tablet_bootstrap.cc:654] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:33.711921 29636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce0b288ab84656b9679b7a7ea321f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:33.713150 30008 log.cc:824] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:33.715189 29845 heartbeater.cc:503] Master 127.16.62.126:44375 was elected leader, sending a full tablet report...
I20260926 23:19:33.716230 30008 tablet_bootstrap.cc:492] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: No bootstrap required, opened a new log
I20260926 23:19:33.716548 30008 ts_tablet_manager.cc:1397] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260926 23:19:33.721755 30010 tablet_bootstrap.cc:492] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: Bootstrap starting.
I20260926 23:19:33.724828 30010 tablet_bootstrap.cc:654] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:33.725956 30010 log.cc:824] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:33.726984 30008 raft_consensus.cc:348] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } }
I20260926 23:19:33.727535 30008 raft_consensus.cc:374] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:33.727648 30008 raft_consensus.cc:729] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63076fa3f11a4c4880890b7fb68de1b2, State: Initialized, Role: FOLLOWER
I20260926 23:19:33.728355 30008 consensus_queue.cc:260] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } }
I20260926 23:19:33.728679 30012 tablet_bootstrap.cc:492] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: Bootstrap starting.
I20260926 23:19:33.729962 30010 tablet_bootstrap.cc:492] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: No bootstrap required, opened a new log
I20260926 23:19:33.730221 29980 heartbeater.cc:503] Master 127.16.62.126:44375 was elected leader, sending a full tablet report...
I20260926 23:19:33.730365 30010 ts_tablet_manager.cc:1397] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 23:19:33.730878 30008 ts_tablet_manager.cc:1428] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 23:19:33.732270 30012 tablet_bootstrap.cc:654] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:33.733796 30012 log.cc:824] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:33.737624 30012 tablet_bootstrap.cc:492] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: No bootstrap required, opened a new log
I20260926 23:19:33.737900 30012 ts_tablet_manager.cc:1397] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260926 23:19:33.740886 30010 raft_consensus.cc:348] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } }
I20260926 23:19:33.741354 30010 raft_consensus.cc:374] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:33.741441 30010 raft_consensus.cc:729] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004558a7227c401badb352d7dedd1b44, State: Initialized, Role: FOLLOWER
I20260926 23:19:33.742225 30010 consensus_queue.cc:260] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } }
I20260926 23:19:33.744042 30010 ts_tablet_manager.cc:1428] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 23:19:33.748409 30012 raft_consensus.cc:348] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } }
I20260926 23:19:33.748912 30012 raft_consensus.cc:374] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:33.749012 30012 raft_consensus.cc:729] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1281a4a912e14744a72944b3453e111d, State: Initialized, Role: FOLLOWER
I20260926 23:19:33.749676 30012 consensus_queue.cc:260] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } }
I20260926 23:19:33.751080 29701 heartbeater.cc:503] Master 127.16.62.126:44375 was elected leader, sending a full tablet report...
I20260926 23:19:33.751562 30012 ts_tablet_manager.cc:1428] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
W20260926 23:19:33.752830 29702 tablet.cc:2369] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:19:33.763851 30013 raft_consensus.cc:482] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:19:33.764155 30013 raft_consensus.cc:504] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } }
I20260926 23:19:33.765853 30013 leader_election.cc:288] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 004558a7227c401badb352d7dedd1b44 (127.16.62.66:35199), 1281a4a912e14744a72944b3453e111d (127.16.62.65:36487)
I20260926 23:19:33.771775 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.766169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:53255 (local address 127.16.62.66:35199)
0926 23:19:33.766371 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.766378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.766623 (+ 245us) server_negotiation.cc:407] Connection header received
0926 23:19:33.766815 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.766825 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.766882 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.766979 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:33.767396 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.768148 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.770169 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.770924 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.771094 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.771166 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.771411 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.771537 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:19:33.772228 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.766196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:35199 (local address 127.16.62.67:53255)
0926 23:19:33.766499 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:33.766518 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:33.766649 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:33.766999 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:33.767004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:33.767033 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:33.767265 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.767273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.768321 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.768326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.769989 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.770000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.771074 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.771080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.771262 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.771284 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:19:33.771387 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 23:19:33.771512 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"spinlock_wait_cycles":20224}
I20260926 23:19:33.772516 30005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.766330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:41979 (local address 127.16.62.65:36487)
0926 23:19:33.766559 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.766567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.767077 (+ 510us) server_negotiation.cc:407] Connection header received
0926 23:19:33.767261 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.767266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.767329 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.767422 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:33.767917 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.768987 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.770744 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.771518 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.771707 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.771774 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.772108 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.772272 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 23:19:33.772599 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.766317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:36487 (local address 127.16.62.67:41979)
0926 23:19:33.766954 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:33.766968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:33.767124 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:33.767432 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:33.767436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:33.767457 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:33.767777 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.767785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.769155 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.769164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.770568 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:33.770579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.771714 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.771720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:33.771945 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.771972 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 23:19:33.772058 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:19:33.772205 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":323,"threads_started":1}
I20260926 23:19:33.773223 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce0b288ab84656b9679b7a7ea321f4" candidate_uuid: "63076fa3f11a4c4880890b7fb68de1b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004558a7227c401badb352d7dedd1b44" is_pre_election: true
I20260926 23:19:33.773547 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce0b288ab84656b9679b7a7ea321f4" candidate_uuid: "63076fa3f11a4c4880890b7fb68de1b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1281a4a912e14744a72944b3453e111d" is_pre_election: true
I20260926 23:19:33.773710 29800 raft_consensus.cc:2454] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63076fa3f11a4c4880890b7fb68de1b2 in term 0.
I20260926 23:19:33.774075 29656 raft_consensus.cc:2454] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63076fa3f11a4c4880890b7fb68de1b2 in term 0.
I20260926 23:19:33.774376 29868 leader_election.cc:302] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004558a7227c401badb352d7dedd1b44, 63076fa3f11a4c4880890b7fb68de1b2; no voters:
I20260926 23:19:33.774762 30013 raft_consensus.cc:2781] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:19:33.774907 30013 raft_consensus.cc:482] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:19:33.775020 30013 raft_consensus.cc:3037] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:33.777247 30013 raft_consensus.cc:504] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } }
I20260926 23:19:33.777772 30013 leader_election.cc:288] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [CANDIDATE]: Term 1 election: Requested vote from peers 004558a7227c401badb352d7dedd1b44 (127.16.62.66:35199), 1281a4a912e14744a72944b3453e111d (127.16.62.65:36487)
I20260926 23:19:33.778194 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce0b288ab84656b9679b7a7ea321f4" candidate_uuid: "63076fa3f11a4c4880890b7fb68de1b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004558a7227c401badb352d7dedd1b44"
I20260926 23:19:33.778205 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce0b288ab84656b9679b7a7ea321f4" candidate_uuid: "63076fa3f11a4c4880890b7fb68de1b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1281a4a912e14744a72944b3453e111d"
I20260926 23:19:33.778383 29800 raft_consensus.cc:3037] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:33.778405 29656 raft_consensus.cc:3037] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:33.780579 29800 raft_consensus.cc:2454] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63076fa3f11a4c4880890b7fb68de1b2 in term 1.
I20260926 23:19:33.780599 29656 raft_consensus.cc:2454] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63076fa3f11a4c4880890b7fb68de1b2 in term 1.
I20260926 23:19:33.781023 29869 leader_election.cc:302] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1281a4a912e14744a72944b3453e111d, 63076fa3f11a4c4880890b7fb68de1b2; no voters:
I20260926 23:19:33.781328 30013 raft_consensus.cc:2781] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:19:33.781744 30013 raft_consensus.cc:686] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 1 LEADER]: Becoming Leader. State: Replica: 63076fa3f11a4c4880890b7fb68de1b2, State: Running, Role: LEADER
I20260926 23:19:33.782230 30013 consensus_queue.cc:237] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } }
I20260926 23:19:33.787017 29503 catalog_manager.cc:5495] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63076fa3f11a4c4880890b7fb68de1b2 (127.16.62.67). New cstate: current_term: 1 leader_uuid: "63076fa3f11a4c4880890b7fb68de1b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63076fa3f11a4c4880890b7fb68de1b2" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 44135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 } health_report { overall_health: UNKNOWN } } }
I20260926 23:19:33.817039 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.811466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.16.62.67:44135)
0926 23:19:33.811676 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.811682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.811799 (+ 117us) server_negotiation.cc:407] Connection header received
0926 23:19:33.812044 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.812067 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.812205 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.812370 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:19:33.812772 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.813476 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.814911 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.815311 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.815445 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.815945 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:19:33.816452 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:19:33.816608 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.816710 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.816787 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 23:19:33.821422 29981 tablet.cc:2369] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:19:33.825254 29656 raft_consensus.cc:1261] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [term 1 FOLLOWER]: Refusing update from remote peer 63076fa3f11a4c4880890b7fb68de1b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:19:33.825289 29800 raft_consensus.cc:1261] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [term 1 FOLLOWER]: Refusing update from remote peer 63076fa3f11a4c4880890b7fb68de1b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:19:33.826202 30013 consensus_queue.cc:1035] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "004558a7227c401badb352d7dedd1b44" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:19:33.826468 30018 consensus_queue.cc:1035] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1281a4a912e14744a72944b3453e111d" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 36487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:19:33.847679 30022 mvcc.cc:205] Tried to move back new op lower bound from 7333743713575219200 to 7333743713412292608. Current Snapshot: MvccSnapshot[applied={T|T < 7333743713575219200 or (T in {7333743713575219200})}]
I20260926 23:19:33.853413 29503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:19:33.853729 29503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:19:33.857702 29503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c5169813f4a4ae48e912d6a9c783add: Sending DeleteTablet for 3 replicas of tablet a3ce0b288ab84656b9679b7a7ea321f4
I20260926 23:19:33.858842 29915 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ce0b288ab84656b9679b7a7ea321f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:19:33 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:42486
I20260926 23:19:33.858918 29780 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ce0b288ab84656b9679b7a7ea321f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:19:33 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:47192
I20260926 23:19:33.859431 29636 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ce0b288ab84656b9679b7a7ea321f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:19:33 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:56228
I20260926 23:19:33.864992 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 29541
I20260926 23:19:33.868804 30033 tablet_replica.cc:330] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: stopping tablet replica
I20260926 23:19:33.869072 30031 tablet_replica.cc:330] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: stopping tablet replica
I20260926 23:19:33.869275 30033 raft_consensus.cc:2229] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:19:33.869622 30031 raft_consensus.cc:2229] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:19:33.869822 30033 raft_consensus.cc:2258] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:19:33.870734 30031 raft_consensus.cc:2258] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:19:33.875463 30033 ts_tablet_manager.cc:1905] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:19:33.876075 30032 tablet_replica.cc:330] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: stopping tablet replica
I20260926 23:19:33.876418 30031 ts_tablet_manager.cc:1905] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:19:33.876487 30032 raft_consensus.cc:2229] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:19:33.876986 30032 raft_consensus.cc:2258] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:19:33.879068 30032 ts_tablet_manager.cc:1905] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:19:33.883677 30033 ts_tablet_manager.cc:1918] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:33.883903 30033 log.cc:1196] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/a3ce0b288ab84656b9679b7a7ea321f4
I20260926 23:19:33.883966 30031 ts_tablet_manager.cc:1918] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:33.884660 30033 ts_tablet_manager.cc:1939] T a3ce0b288ab84656b9679b7a7ea321f4 P 1281a4a912e14744a72944b3453e111d: Deleting consensus metadata
I20260926 23:19:33.886035 30032 ts_tablet_manager.cc:1918] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:33.886207 30032 log.cc:1196] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/a3ce0b288ab84656b9679b7a7ea321f4
I20260926 23:19:33.886739 30032 ts_tablet_manager.cc:1939] T a3ce0b288ab84656b9679b7a7ea321f4 P 004558a7227c401badb352d7dedd1b44: Deleting consensus metadata
I20260926 23:19:33.886984 29490 catalog_manager.cc:4841] TS 1281a4a912e14744a72944b3453e111d (127.16.62.65:36487): tablet a3ce0b288ab84656b9679b7a7ea321f4 (table test-table [id=5719320eb3244aab9e981c327b0f159f]) successfully deleted
I20260926 23:19:33.884131 30031 log.cc:1196] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/a3ce0b288ab84656b9679b7a7ea321f4
I20260926 23:19:33.888324 30031 ts_tablet_manager.cc:1939] T a3ce0b288ab84656b9679b7a7ea321f4 P 63076fa3f11a4c4880890b7fb68de1b2: Deleting consensus metadata
I20260926 23:19:33.888983 29489 catalog_manager.cc:4841] TS 004558a7227c401badb352d7dedd1b44 (127.16.62.66:35199): tablet a3ce0b288ab84656b9679b7a7ea321f4 (table test-table [id=5719320eb3244aab9e981c327b0f159f]) successfully deleted
I20260926 23:19:33.890322 29489 catalog_manager.cc:4841] TS 63076fa3f11a4c4880890b7fb68de1b2 (127.16.62.67:44135): tablet a3ce0b288ab84656b9679b7a7ea321f4 (table test-table [id=5719320eb3244aab9e981c327b0f159f]) successfully deleted
I20260926 23:19:33.894776 30005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:33.866563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.16.62.65:36487)
0926 23:19:33.866778 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:33.866787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:33.866812 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:19:33.866888 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:33.866896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:33.867025 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:33.867172 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:33.872701 (+ 5529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.874174 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.876068 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:33.876765 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:33.876887 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:33.887806 (+ 10919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:33.887821 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:33.887840 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:33.887892 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:33.891588 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:33.892610 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:33.892618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:33.892620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:33.892765 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:33.893198 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:33.893202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:33.893204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:33.893580 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:33.893824 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:33.894214 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 23:19:33.894388 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:19:33.895535 29696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:33.988042 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 29541
I20260926 23:19:34.005861 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 29705
I20260926 23:19:34.026682 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:34.006766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.16.62.66:35199)
0926 23:19:34.006970 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:34.006976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:34.006999 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:19:34.007069 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:34.007075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:34.007156 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:34.007265 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:34.008580 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:34.009757 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:34.011129 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:34.011794 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:34.011923 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:34.019084 (+ 7161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:34.019105 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:34.019128 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:34.019183 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:34.023367 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:34.024188 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:34.024198 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:34.024201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:34.024363 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:34.024782 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:34.024787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:34.024791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:34.025240 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:34.025547 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:34.025850 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 23:19:34.026094 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:19:34.027485 29840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:34.119737 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 29705
I20260926 23:19:34.137360 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 29849
I20260926 23:19:34.153614 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:34.138240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42500 (local address 127.16.62.67:44135)
0926 23:19:34.138496 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:34.138502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:34.138529 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:19:34.138603 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:34.138614 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:34.138736 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:34.138845 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:34.140042 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:34.141312 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:34.142677 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:34.143376 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:34.143517 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:34.146958 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:34.146980 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:34.146995 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:34.147036 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:34.150323 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:34.151260 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:34.151267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:34.151269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:34.151453 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:34.151895 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:34.151899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:34.151901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:34.152417 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:34.152666 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:34.153057 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 23:19:34.153257 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 23:19:34.154390 29975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:34.278867 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 29849
I20260926 23:19:34.299031 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 29471
I20260926 23:19:34.300412 29534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:34.417532 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 29471
I20260926 23:19:34.438758 16633 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 23:19:34.775053 16633 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 23:19:34.775234 16633 mini_ranger.cc:66] Stopping Ranger...
I20260926 23:19:35.121273 16633 mini_ranger.cc:68] Stopped Ranger
2026-09-26 23:19:35.121 UTC [28523] LOG: received smart shutdown request
2026-09-26 23:19:35.125 UTC [28523] LOG: background worker "logical replication launcher" (PID 28532) exited with exit code 1
2026-09-26 23:19:35.125 UTC [28527] LOG: shutting down
2026-09-26 23:19:35.310 UTC [28523] LOG: database system is shut down
2026-09-26T23:19:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52288 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:19:35 dist-test-slave-vcv1 krb5kdc[30046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:19:35 dist-test-slave-vcv1 krb5kdc[30046](info): set up 2 sockets
Sep 26 23:19:35 dist-test-slave-vcv1 krb5kdc[30046](info): commencing operation
krb5kdc: starting...
W20260926 23:19:36.569541 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:19:36 dist-test-slave-vcv1 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:19:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:19:36.807898 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:35495
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:35495,127.16.62.125:35239,127.16.62.124:34861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:37.136881 30059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:37.137214 30059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:37.137411 30059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:37.144740 30059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:19:37.144860 30059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:37.144974 30059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:37.145072 30059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:19:37.145123 30059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:19:37.155113 30059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:35495,127.16.62.125:35239,127.16.62.124:34861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:35495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:37.156598 30059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:37.157954 30059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:37.170193 30064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:37.170284 30067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:37.170150 30065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:37.171310 30059 server_base.cc:1034] running on GCE node
I20260926 23:19:37.172118 30059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:37.173601 30059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:37.174804 30059 hybrid_clock.cc:648] HybridClock initialized: now 1790464777174755 us; error 36 us; skew 500 ppm
I20260926 23:19:37.175160 30059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:37 dist-test-slave-vcv1 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:37.182904 30059 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:19:37.184566 30059 webserver.cc:460] Webserver started at http://127.16.62.126:45791/ using document root <none> and password file <none>
I20260926 23:19:37.185271 30059 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:37.185398 30059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:37.185724 30059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:19:37.188771 30059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "d1ffda901b4a450baace2ec0266c45ce"
format_stamp: "Formatted at 2026-09-26 23:19:37 on dist-test-slave-vcv1"
server_key: "165a8e6fe3d0e0100401f88d15d39c3b"
server_key_iv: "0abb639aa0bf078c2801213c6bf207a7"
server_key_version: "encryptionkey@0"
I20260926 23:19:37.189563 30059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "d1ffda901b4a450baace2ec0266c45ce"
format_stamp: "Formatted at 2026-09-26 23:19:37 on dist-test-slave-vcv1"
server_key: "165a8e6fe3d0e0100401f88d15d39c3b"
server_key_iv: "0abb639aa0bf078c2801213c6bf207a7"
server_key_version: "encryptionkey@0"
I20260926 23:19:37.194756 30059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:19:37.198289 30074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:37.199679 30059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 23:19:37.199858 30059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "d1ffda901b4a450baace2ec0266c45ce"
format_stamp: "Formatted at 2026-09-26 23:19:37 on dist-test-slave-vcv1"
server_key: "165a8e6fe3d0e0100401f88d15d39c3b"
server_key_iv: "0abb639aa0bf078c2801213c6bf207a7"
server_key_version: "encryptionkey@0"
I20260926 23:19:37.200034 30059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:37.236272 30059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:37.240892 30059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:37.241209 30059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:37.260982 30059 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:35495
I20260926 23:19:37.261025 30125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:35495 every 8 connection(s)
I20260926 23:19:37.262506 30059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:19:37.266332 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:37.271063 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 30059
I20260926 23:19:37.271301 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:19:37.272230 16633 external_mini_cluster.cc:1428] Setting key 3c70a445c9faca3a2e2bd2a73ff9b611
I20260926 23:19:37.274206 30126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:19:37 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.62.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.125@KRBTEST.COM" created.
I20260926 23:19:37.301156 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.278375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35495 (local address 127.0.0.1:43430)
0926 23:19:37.279378 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.279430 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:37.282067 (+ 2637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:37.283086 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:37.283092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:37.283670 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:37.284027 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:37.284045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.286440 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.286444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:37.287595 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:37.287609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.289044 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.289050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:37.289163 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:37.289957 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:37.289981 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:37.292569 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:37.296971 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:37.296990 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:37.296999 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:37.297433 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:37.297949 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:37.297953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:37.297956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:37.298180 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:37.299156 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:37.299162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:37.299368 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 23:19:37.299590 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 23:19:37.299989 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":440,"threads_started":1}
I20260926 23:19:37.301183 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.278375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.16.62.126:35495)
0926 23:19:37.280453 (+ 2078us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:37.280459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:37.280502 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:19:37.282459 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:37.282470 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:37.282802 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:37.283021 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:37.284281 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.286268 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.287745 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.288781 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.289017 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:37.292847 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:37.292862 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:37.292884 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:37.292935 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:37.296818 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:37.297614 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:37.297621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:37.297623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:37.297782 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:37.298322 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:37.298326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:37.298328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:37.298976 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:37.299179 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:37.299787 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 23:19:37.300222 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":1878,"spinlock_wait_cycles":83840,"thread_start_us":1636,"threads_started":1}
I20260926 23:19:37.304473 30126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:19:37.305545 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.305176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56672)
0926 23:19:37.305321 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.305435 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 23:19:37.305784 30075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
I20260926 23:19:37.307381 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.307001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56674)
0926 23:19:37.307179 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.307243 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260926 23:19:37.307619 30126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:35239: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab.
Entry for principal kudu/127.16.62.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab.
I20260926 23:19:37.328645 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.125:35239
--webserver_interface=127.16.62.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab
--principal=kudu/127.16.62.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:35495,127.16.62.125:35239,127.16.62.124:34861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:19:37.356984 30126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } attempt: 1
I20260926 23:19:37.358276 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.357738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56690)
0926 23:19:37.357965 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.358102 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 23:19:37.359297 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.358902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56704)
0926 23:19:37.359057 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.359139 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 23:19:37.359534 30126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:35239: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
I20260926 23:19:37.427830 30126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } attempt: 2
I20260926 23:19:37.429234 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.428642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56720)
0926 23:19:37.428929 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.429059 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 23:19:37.430380 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.429956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56722)
0926 23:19:37.430121 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.430226 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:19:37.430593 30126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:35239: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
I20260926 23:19:37.521880 30126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } attempt: 3
I20260926 23:19:37.523208 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.522634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56732)
0926 23:19:37.522891 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.523023 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 23:19:37.524349 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.523907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56736)
0926 23:19:37.524058 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.524184 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 23:19:37.524600 30126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:35239: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
W20260926 23:19:37.657783 30133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:37.658072 30133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:37.658229 30133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:37.665640 30133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:19:37.665755 30133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:37.665839 30133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:37.665940 30133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:19:37.666028 30133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:19:37.667874 30126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } attempt: 4
I20260926 23:19:37.669193 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.668581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56750)
0926 23:19:37.668919 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.669024 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260926 23:19:37.670498 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.670044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56756)
0926 23:19:37.670232 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.670348 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:19:37.670789 30126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:35239: Network error: Client connection negotiation failed: client connection to 127.16.62.125:35239: connect: Connection refused (error 111)
I20260926 23:19:37.677209 30133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:35495,127.16.62.125:35239,127.16.62.124:34861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab
--principal=kudu/127.16.62.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.125:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.62.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:37.678534 30133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:37.679848 30133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:37.691733 30138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:37.691805 30141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:37.691841 30139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:37.692619 30133 server_base.cc:1034] running on GCE node
I20260926 23:19:37.693431 30133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:37.694859 30133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:37.696089 30133 hybrid_clock.cc:648] HybridClock initialized: now 1790464777696044 us; error 41 us; skew 500 ppm
I20260926 23:19:37.696527 30133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:37 dist-test-slave-vcv1 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:37.699926 30133 init.cc:377] Logged in from keytab as kudu/127.16.62.125@KRBTEST.COM (short username kudu)
I20260926 23:19:37.701638 30133 webserver.cc:460] Webserver started at http://127.16.62.125:43943/ using document root <none> and password file <none>
I20260926 23:19:37.702358 30133 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:37.702467 30133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:37.702744 30133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:19:37.706230 30133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/data/instance:
uuid: "2cb2208698064486a07fb3122168e9f7"
format_stamp: "Formatted at 2026-09-26 23:19:37 on dist-test-slave-vcv1"
server_key: "97b1bc3bf740f08d432b7b31b569bbe3"
server_key_iv: "28a5610b7e40880e5d5f22dbf385f434"
server_key_version: "encryptionkey@0"
I20260926 23:19:37.707046 30133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/wal/instance:
uuid: "2cb2208698064486a07fb3122168e9f7"
format_stamp: "Formatted at 2026-09-26 23:19:37 on dist-test-slave-vcv1"
server_key: "97b1bc3bf740f08d432b7b31b569bbe3"
server_key_iv: "28a5610b7e40880e5d5f22dbf385f434"
server_key_version: "encryptionkey@0"
I20260926 23:19:37.712441 30133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 23:19:37.715989 30148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:37.717351 30133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:19:37.717530 30133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/wal
uuid: "2cb2208698064486a07fb3122168e9f7"
format_stamp: "Formatted at 2026-09-26 23:19:37 on dist-test-slave-vcv1"
server_key: "97b1bc3bf740f08d432b7b31b569bbe3"
server_key_iv: "28a5610b7e40880e5d5f22dbf385f434"
server_key_version: "encryptionkey@0"
I20260926 23:19:37.717746 30133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:37.734637 30133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:37.739604 30133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:37.739946 30133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:37.762863 30133 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.125:35239
I20260926 23:19:37.762918 30199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.125:35239 every 8 connection(s)
I20260926 23:19:37.764319 30133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
I20260926 23:19:37.768103 30200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:37.772645 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 30133
I20260926 23:19:37.772845 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-1/wal/instance
I20260926 23:19:37.773298 16633 external_mini_cluster.cc:1428] Setting key bd9b9611dd6adaa76901511b9f4391c9
I20260926 23:19:37.774852 30200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:19:37 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.62.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.124@KRBTEST.COM" created.
I20260926 23:19:37.797607 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.777530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.16.62.126:35495)
0926 23:19:37.777751 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:37.777758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:37.778930 (+ 1172us) server_negotiation.cc:407] Connection header received
0926 23:19:37.781126 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:37.781131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:37.781198 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:37.781297 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:37.783339 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.784018 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.786516 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.786935 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.787031 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:37.791155 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:37.791161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:37.791164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:37.791197 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:37.793861 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:37.794908 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:37.794915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:37.794940 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:37.795132 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:37.795623 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:37.795627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:37.795629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:37.796044 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:37.796257 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:37.797054 (+ 797us) server_negotiation.cc:299] Negotiation successful
0926 23:19:37.797335 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:19:37.798614 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.777915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35495 (local address 127.0.0.1:43444)
0926 23:19:37.778753 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.778798 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:37.780818 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:37.781552 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:37.781565 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:37.782327 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:37.783130 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:37.783158 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.784215 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.784224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:37.786347 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:37.786370 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.787161 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.787170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:37.787480 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:37.788254 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:37.788287 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:37.790918 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:37.794009 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:37.794018 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:37.794034 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:37.794722 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:37.795272 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:37.795280 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:37.795284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:37.795516 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:37.796262 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:37.796270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:37.796606 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 23:19:37.796846 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 23:19:37.797414 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":349,"threads_started":1}
I20260926 23:19:37.800669 30200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:19:37 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:19:37.815109 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.801359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56770)
0926 23:19:37.801502 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.801524 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:37.801633 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:37.802732 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:37.802737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:37.803236 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:37.803366 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:37.803374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.804873 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.804881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:37.805712 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:37.805722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.806508 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.806513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:37.806748 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:37.807358 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:37.807375 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:37.809410 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:37.812659 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:37.812664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:37.812668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:37.813025 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:37.813398 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:37.813404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:37.813407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:37.813558 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:37.814123 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:37.814128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:37.814266 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 23:19:37.814412 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 23:19:37.814703 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260926 23:19:37.815214 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.801699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.16.62.125:35239)
0926 23:19:37.802345 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:37.802350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:37.802374 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:19:37.802414 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:37.802420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:37.802594 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:37.802744 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:37.803519 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.804746 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.805845 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.806367 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.806503 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:37.809620 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:37.809632 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:37.809644 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:37.809672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:37.812478 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:37.813163 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:37.813171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:37.813174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:37.813290 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:37.813659 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:37.813667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:37.813670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:37.813994 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:37.814218 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:37.814565 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 23:19:37.814816 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":501,"thread_start_us":372,"threads_started":1}
I20260926 23:19:37.817716 30200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:19:37.818959 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.818488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59366)
0926 23:19:37.818692 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.818796 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 23:19:37.819284 30151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
I20260926 23:19:37.821193 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.820843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59376)
0926 23:19:37.820992 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.821075 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 23:19:37.821424 30200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:34861: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab.
Entry for principal kudu/127.16.62.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab.
I20260926 23:19:37.828033 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.124:34861
--webserver_interface=127.16.62.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab
--principal=kudu/127.16.62.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:35495,127.16.62.125:35239,127.16.62.124:34861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:19:37.870800 30200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } attempt: 1
I20260926 23:19:37.872035 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.871551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59380)
0926 23:19:37.871763 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.871881 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 23:19:37.873083 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.872727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59396)
0926 23:19:37.872906 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.872976 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:19:37.873301 30200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:34861: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
I20260926 23:19:37.941537 30200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } attempt: 2
I20260926 23:19:37.942740 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.942275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59404)
0926 23:19:37.942492 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.942593 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 23:19:37.943761 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.943416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59416)
0926 23:19:37.943578 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.943642 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 23:19:37.943966 30200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:34861: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
I20260926 23:19:37.970196 30126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } attempt: 5
Sep 26 23:19:37 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:19:37.983693 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.971082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.16.62.125:35239)
0926 23:19:37.971289 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:37.971296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:37.971320 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:19:37.971415 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:37.971421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:37.971518 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:37.971628 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:37.972508 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.973611 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.974816 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.975290 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.975429 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:37.978463 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:37.978468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:37.978472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:37.978504 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:37.981182 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:37.981828 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:37.981841 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:37.981843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:37.981957 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:37.982326 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:37.982333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:37.982336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:37.982615 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:37.982769 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:37.983151 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 23:19:37.983365 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:19:37.983704 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.970955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56776)
0926 23:19:37.971178 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.971207 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:37.971315 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:37.971697 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:37.971700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:37.972051 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:37.972315 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:37.972325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.973757 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.973761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:37.974665 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:37.974678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:37.975451 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:37.975457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:37.975638 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:37.976289 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:37.976307 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:37.978277 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:37.981323 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:37.981330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:37.981337 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:37.981703 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:37.982068 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:37.982073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:37.982077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:37.982216 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:37.982735 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:37.982740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:37.982890 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 23:19:37.983034 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 23:19:37.983338 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 23:19:37.985039 30126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:19:37.986145 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.985763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59418)
0926 23:19:37.985930 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.986006 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 23:19:37.987044 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:37.986744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59426)
0926 23:19:37.986875 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:37.986957 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 23:19:37.987270 30126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:34861: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
I20260926 23:19:38.035216 30200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } attempt: 3
I20260926 23:19:38.036638 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.036073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59442)
0926 23:19:38.036343 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.036457 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 23:19:38.037779 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.037347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59458)
0926 23:19:38.037574 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.037667 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260926 23:19:38.037998 30200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:34861: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
I20260926 23:19:38.038480 30126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } attempt: 1
I20260926 23:19:38.039547 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.039052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59468)
0926 23:19:38.039257 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.039367 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 23:19:38.040617 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.040236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59480)
0926 23:19:38.040385 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.040480 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 23:19:38.040844 30126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:34861: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
I20260926 23:19:38.109081 30126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } attempt: 2
I20260926 23:19:38.110356 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.109761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59486)
0926 23:19:38.110015 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.110127 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 23:19:38.111475 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.111054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59492)
0926 23:19:38.111230 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.111336 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:19:38.111712 30126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:34861: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
W20260926 23:19:38.158540 30207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:38.158812 30207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:38.158902 30207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:38.166256 30207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:19:38.166359 30207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:38.166406 30207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:38.166463 30207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:19:38.166507 30207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:19:38.176183 30207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:35495,127.16.62.125:35239,127.16.62.124:34861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab
--principal=kudu/127.16.62.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.124:34861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.62.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:38.177388 30207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:38.178617 30207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:19:38.181206 30200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } attempt: 4
I20260926 23:19:38.182446 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.181869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59502)
0926 23:19:38.182134 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.182241 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 23:19:38.183634 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.183202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59504)
0926 23:19:38.183407 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.183510 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 23:19:38.183970 30200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:34861: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
W20260926 23:19:38.190755 30212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:38.190716 30213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:38.190755 30215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:38.191706 30207 server_base.cc:1034] running on GCE node
I20260926 23:19:38.192553 30207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:38.193934 30207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:38.195195 30207 hybrid_clock.cc:648] HybridClock initialized: now 1790464778195127 us; error 69 us; skew 500 ppm
I20260926 23:19:38.195605 30207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:38 dist-test-slave-vcv1 krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464778, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:38.198906 30207 init.cc:377] Logged in from keytab as kudu/127.16.62.124@KRBTEST.COM (short username kudu)
I20260926 23:19:38.200357 30207 webserver.cc:460] Webserver started at http://127.16.62.124:41013/ using document root <none> and password file <none>
I20260926 23:19:38.200990 30207 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:38.201084 30207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:38.201375 30207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:19:38.204576 30207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/data/instance:
uuid: "774a6300796b41bfbcc76bb201ffafd0"
format_stamp: "Formatted at 2026-09-26 23:19:38 on dist-test-slave-vcv1"
server_key: "02eb8ae9c0cecb084b68e2f2794ac935"
server_key_iv: "b40b50101a6a6f74626390fc43f37188"
server_key_version: "encryptionkey@0"
I20260926 23:19:38.205372 30207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/wal/instance:
uuid: "774a6300796b41bfbcc76bb201ffafd0"
format_stamp: "Formatted at 2026-09-26 23:19:38 on dist-test-slave-vcv1"
server_key: "02eb8ae9c0cecb084b68e2f2794ac935"
server_key_iv: "b40b50101a6a6f74626390fc43f37188"
server_key_version: "encryptionkey@0"
I20260926 23:19:38.211207 30207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260926 23:19:38.215073 30222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:38.216554 30207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:19:38.216699 30207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/wal
uuid: "774a6300796b41bfbcc76bb201ffafd0"
format_stamp: "Formatted at 2026-09-26 23:19:38 on dist-test-slave-vcv1"
server_key: "02eb8ae9c0cecb084b68e2f2794ac935"
server_key_iv: "b40b50101a6a6f74626390fc43f37188"
server_key_version: "encryptionkey@0"
I20260926 23:19:38.216879 30207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:38.217926 30126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } attempt: 3
I20260926 23:19:38.218928 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.218503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59512)
0926 23:19:38.218687 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.218779 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 23:19:38.219928 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.219560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59518)
0926 23:19:38.219760 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.219828 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 23:19:38.220207 30126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:34861: Network error: Client connection negotiation failed: client connection to 127.16.62.124:34861: connect: Connection refused (error 111)
I20260926 23:19:38.231616 30207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:38.235983 30207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:38.236311 30207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:38.255570 30207 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.124:34861
I20260926 23:19:38.255600 30273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.124:34861 every 8 connection(s)
I20260926 23:19:38.257035 30207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
I20260926 23:19:38.260021 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 30207
I20260926 23:19:38.260315 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464575313384-16633-0/minicluster-data/master-2/wal/instance
I20260926 23:19:38.260927 16633 external_mini_cluster.cc:1428] Setting key 28c1a0c3eae4e1226142c8d85360e31f
I20260926 23:19:38.261575 30274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:38.274873 30274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:19:38 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464778, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
Sep 26 23:19:38 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
I20260926 23:19:38.295252 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.278387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.16.62.126:35495)
0926 23:19:38.278635 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.278646 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.279225 (+ 579us) server_negotiation.cc:407] Connection header received
0926 23:19:38.279540 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.279545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.279630 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.279763 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.281053 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.281890 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.283648 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.284403 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.284577 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.288456 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.288467 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.288486 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.288540 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.291730 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.292688 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.292697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.292701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.292854 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.293333 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.293343 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.293345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.293625 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.293802 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.294587 (+ 785us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.294826 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 23:19:38.296309 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.262986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.16.62.124:34861)
0926 23:19:38.276744 (+ 13758us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.276756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.276796 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:19:38.278785 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.278824 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.279301 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.279548 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.281303 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.282749 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.284766 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.285523 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.285670 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.290056 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.290063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.290076 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.290105 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.292870 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.293540 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.293547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.293549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.293663 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.294057 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.294061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.294063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.294436 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.294726 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.295073 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.295603 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13515,"thread_start_us":13247,"threads_started":1}
I20260926 23:19:38.296356 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.278351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35495 (local address 127.0.0.1:43458)
0926 23:19:38.279057 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.279105 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:38.279271 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:38.279833 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:38.279839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:38.280493 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:38.280873 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.280889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.282050 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.282056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.283499 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.283510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.284666 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.284672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.284987 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.285856 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:38.285877 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:38.288239 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:38.291891 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.291904 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.291917 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.292534 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.292987 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.292994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.292997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.293198 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.293780 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:38.293789 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:38.294152 (+ 363us) client_negotiation.cc:769] Sending connection context
0926 23:19:38.294464 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 23:19:38.294970 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":334,"threads_started":1}
I20260926 23:19:38.298765 30274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:19:38 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464778, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
Sep 26 23:19:38 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:38.313406 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.299509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.16.62.125:35239)
0926 23:19:38.299674 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.299680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.300124 (+ 444us) server_negotiation.cc:407] Connection header received
0926 23:19:38.300282 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.300287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.300374 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.300489 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.301115 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.302039 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.303400 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.304101 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.304255 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.307611 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.307616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.307619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.307652 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.310851 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.311465 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.311470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.311473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.311628 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.311998 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.312006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.312011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.312380 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.312543 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.312934 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.313153 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:19:38.313475 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.299391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:35239 (local address 127.0.0.1:56792)
0926 23:19:38.300028 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.300046 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:38.300163 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:38.300513 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:38.300516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:38.300836 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:38.300973 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.300981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.302170 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.302175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.303265 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.303276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.304280 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.304286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.304478 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.305063 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:38.305079 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:38.307436 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:38.310985 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.310988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.310992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.311349 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.311744 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.311747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.311749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.311886 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.312527 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:38.312531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:38.312669 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 23:19:38.312801 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 23:19:38.313087 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532}
I20260926 23:19:38.314420 30274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:19:38.316219 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.299492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.16.62.126:35495)
0926 23:19:38.299648 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.299654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.299746 (+ 92us) server_negotiation.cc:407] Connection header received
0926 23:19:38.299931 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.299936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.300007 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.300108 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.301706 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.302640 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.304217 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.304603 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.304782 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.309368 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.309375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.309380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.309418 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.312925 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.313928 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.313935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.313939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.314126 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.314546 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.314551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.314554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.314906 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.315241 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.315531 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.315792 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 23:19:38 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464778, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
Sep 26 23:19:38 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:19:38.329202 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.315329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59542)
0926 23:19:38.315532 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.315566 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:38.315661 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:38.316073 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:38.316076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:38.316560 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:38.316685 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.316692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.318001 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.318007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.319041 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.319052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.319907 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.319913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.320107 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.320755 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:38.320779 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:38.323311 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:38.326523 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.326529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.326536 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.326903 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.327280 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.327288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.327294 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.327430 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.327962 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:38.327967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:38.328123 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 23:19:38.328347 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 23:19:38.328698 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":3}
I20260926 23:19:38.329201 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.315546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.16.62.124:34861)
0926 23:19:38.315703 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.315709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.315739 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:19:38.315787 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.315792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.315855 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.315947 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.316812 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.317656 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.319189 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.319772 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.319858 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.323532 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.323537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.323540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.323575 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.326380 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.327025 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.327030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.327035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.327168 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.327545 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.327551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.327555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.327843 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.328001 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.328424 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.328846 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":47}
I20260926 23:19:38.333089 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.317867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.16.62.125:35239)
0926 23:19:38.318062 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.318068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.318114 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:19:38.318162 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.318166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.318232 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.318325 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.319602 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.320683 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.322288 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.322822 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.322947 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.327154 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.327159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.327162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.327192 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.330123 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.330973 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.330978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.330981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.331169 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.331547 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.331552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.331555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.331887 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.332214 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.332499 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.332738 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 23:19:38.335163 30274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0: Bootstrap starting.
I20260926 23:19:38.338017 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:19:38.338340 30274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:38.339287 30274 log.cc:824] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:38.343066 30274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0: No bootstrap required, opened a new log
I20260926 23:19:38.355984 30274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } }
I20260926 23:19:38.356734 30274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:38.356926 30274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774a6300796b41bfbcc76bb201ffafd0, State: Initialized, Role: FOLLOWER
I20260926 23:19:38.357946 30274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } }
I20260926 23:19:38.365244 30287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.367890 30287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:19:38.367762 30274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:19:38.368835 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.346387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.16.62.126:35495)
0926 23:19:38.346652 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.346661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.346688 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:19:38.346934 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.346943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.347051 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.347129 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.349847 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.351133 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.353503 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.354273 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.354379 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.359026 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.359038 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.359046 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.359105 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.364281 (+ 5176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.365518 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.365524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.365528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.365711 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.366640 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.366645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.366649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.367060 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.367313 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.368122 (+ 809us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.368434 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 23:19:38.380501 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.345954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.16.62.124:34861)
0926 23:19:38.346155 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.346164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.357826 (+ 11662us) server_negotiation.cc:407] Connection header received
0926 23:19:38.358385 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.358395 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.358514 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.358622 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.360977 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.362466 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.365947 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.366796 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.366910 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.370519 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.370527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.370536 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.370588 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.375930 (+ 5342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.377005 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.377024 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.377029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.377268 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.377917 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.377926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.377932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.378475 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.379702 (+ 1227us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.379818 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.379979 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 23:19:38.380677 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.347761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.16.62.125:35239)
0926 23:19:38.347996 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.348013 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.364790 (+ 16777us) server_negotiation.cc:407] Connection header received
0926 23:19:38.364964 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.364970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.365065 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.365153 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.367023 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.368183 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.370475 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.370946 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.371073 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.373937 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.373946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.373954 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.374001 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.377160 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.378313 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.378320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.378324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.378507 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.378988 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.378996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.378999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.379423 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.379683 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.380062 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.380323 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 23:19:38.386090 30298 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:19:38.386303 30298 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:19:38.382539 30207 master_runner.cc:186] Error getting master registration for 127.16.62.126:35495: OK, instance_id { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" instance_seqno: 1790464777255151 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:19:38.394347 30207 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:19:38.397425 30126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } attempt: 4
Sep 26 23:19:38 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
I20260926 23:19:38.413244 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.397916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59558)
0926 23:19:38.398142 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.398165 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:38.398282 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:38.398608 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:38.398614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:38.399165 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:38.399398 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.399413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.400600 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.400607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.402237 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.402256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.403405 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.403410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.403587 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.404315 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:38.404343 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:38.406330 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:38.410477 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.410481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.410483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.410827 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.411244 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.411247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.411249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.411367 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.412101 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:38.412107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:38.412336 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 23:19:38.412505 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 23:19:38.412804 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 23:19:38.413455 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.398054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.16.62.124:34861)
0926 23:19:38.398193 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.398198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.398213 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:19:38.398404 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.398407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.398460 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.398534 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.399552 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.400449 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.402431 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.403252 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.403400 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.406510 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.406517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.406523 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.406563 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.410329 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.410942 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.410949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.410953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.411131 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.411473 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.411478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.411482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.411868 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.412028 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.412838 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.413076 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 23:19:38.417701 30126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce: Bootstrap starting.
I20260926 23:19:38.421784 30126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:38.422991 30126 log.cc:824] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:38.425673 30126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce: No bootstrap required, opened a new log
I20260926 23:19:38.433513 30126 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } }
I20260926 23:19:38.433861 30126 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:38.433991 30126 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1ffda901b4a450baace2ec0266c45ce, State: Initialized, Role: FOLLOWER
I20260926 23:19:38.434537 30126 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } }
I20260926 23:19:38.435482 30301 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.435905 30301 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:19:38.436805 30126 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:19:38.447851 30312 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:19:38.448021 30312 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:19:38.446877 30059 master_runner.cc:186] Error getting master registration for 127.16.62.125:35239: OK, instance_id { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" instance_seqno: 1790464777757159 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:19:38.455380 30059 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:19:38.483206 30200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } attempt: 5
W20260926 23:19:38.488498 30228 tablet.cc:2369] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 23:19:38 dist-test-slave-vcv1 krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
W20260926 23:19:38.494194 30080 tablet.cc:2369] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:19:38.501192 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.483885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:34861 (local address 127.0.0.1:59570)
0926 23:19:38.484196 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:38.484218 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:38.484385 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:38.484756 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:38.484761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:38.485279 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:38.485452 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.485461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.486579 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.486588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.487770 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:38.487784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.489109 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.489118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:38.489367 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.490081 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:38.490101 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:38.492326 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:38.498409 (+ 6083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.498413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.498416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.498829 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.499240 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:38.499249 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:38.499252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:38.499402 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:38.500001 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:38.500007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:38.500234 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 23:19:38.500415 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 23:19:38.500741 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 23:19:38.501222 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:38.483962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.16.62.124:34861)
0926 23:19:38.484194 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:38.484201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:38.484306 (+ 105us) server_negotiation.cc:407] Connection header received
0926 23:19:38.484527 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:38.484538 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:38.484608 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:38.484730 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:38.485624 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.486417 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.487936 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:38.488702 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:38.488857 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:38.492562 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:38.492568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:38.492571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:38.492605 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:38.498230 (+ 5625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.498962 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.498970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.498974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.499114 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:38.499527 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:38.499533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:38.499540 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:38.499862 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:38.500028 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:38.500546 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 23:19:38.500823 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:19:38.506445 30200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7: Bootstrap starting.
I20260926 23:19:38.509763 30200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:38.510848 30200 log.cc:824] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:38.513702 30200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7: No bootstrap required, opened a new log
I20260926 23:19:38.523940 30200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } }
I20260926 23:19:38.524420 30200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:38.524564 30200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb2208698064486a07fb3122168e9f7, State: Initialized, Role: FOLLOWER
I20260926 23:19:38.525238 30200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } }
I20260926 23:19:38.526294 30315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.526785 30315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:19:38.527552 30200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:19:38.547163 30133 master_runner.cc:263] No leader master found from master 2cb2208698064486a07fb3122168e9f7
I20260926 23:19:38.547348 30133 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 23:19:38.549332 30326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:19:38.549489 30326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:19:38.580523 30315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:19:38.580852 30315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } }
I20260926 23:19:38.582576 30315 leader_election.cc:288] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1ffda901b4a450baace2ec0266c45ce (127.16.62.126:35495), 774a6300796b41bfbcc76bb201ffafd0 (127.16.62.124:34861)
I20260926 23:19:38.583963 30101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cb2208698064486a07fb3122168e9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1ffda901b4a450baace2ec0266c45ce" is_pre_election: true
I20260926 23:19:38.584656 30101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb2208698064486a07fb3122168e9f7 in term 0.
I20260926 23:19:38.585371 30249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cb2208698064486a07fb3122168e9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774a6300796b41bfbcc76bb201ffafd0" is_pre_election: true
I20260926 23:19:38.585973 30249 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb2208698064486a07fb3122168e9f7 in term 0.
I20260926 23:19:38.586957 30151 leader_election.cc:302] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb2208698064486a07fb3122168e9f7, 774a6300796b41bfbcc76bb201ffafd0; no voters:
I20260926 23:19:38.587893 30315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:19:38.588071 30315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:19:38.588224 30315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:38.591662 30315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } }
I20260926 23:19:38.592834 30101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cb2208698064486a07fb3122168e9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1ffda901b4a450baace2ec0266c45ce"
I20260926 23:19:38.593084 30101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:38.596851 30101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb2208698064486a07fb3122168e9f7 in term 1.
I20260926 23:19:38.597368 30149 leader_election.cc:302] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb2208698064486a07fb3122168e9f7, d1ffda901b4a450baace2ec0266c45ce; no voters:
I20260926 23:19:38.601178 30328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:19:38.602674 30249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cb2208698064486a07fb3122168e9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774a6300796b41bfbcc76bb201ffafd0"
I20260926 23:19:38.602924 30249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:38.606554 30249 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb2208698064486a07fb3122168e9f7 in term 1.
I20260926 23:19:38.607125 30315 leader_election.cc:288] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [CANDIDATE]: Term 1 election: Requested vote from peers d1ffda901b4a450baace2ec0266c45ce (127.16.62.126:35495), 774a6300796b41bfbcc76bb201ffafd0 (127.16.62.124:34861)
I20260926 23:19:38.608284 30328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb2208698064486a07fb3122168e9f7, State: Running, Role: LEADER
I20260926 23:19:38.609006 30328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } }
I20260926 23:19:38.619645 30315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cb2208698064486a07fb3122168e9f7. Latest consensus state: current_term: 1 leader_uuid: "2cb2208698064486a07fb3122168e9f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.619958 30315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:38.633016 30331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:19:38.636652 30331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:19:38.657222 30249 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [term 1 FOLLOWER]: Refusing update from remote peer 2cb2208698064486a07fb3122168e9f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:19:38.658406 30315 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:19:38.667521 30101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [term 1 FOLLOWER]: Refusing update from remote peer 2cb2208698064486a07fb3122168e9f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:19:38.668591 30315 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:19:38.705219 30301 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cb2208698064486a07fb3122168e9f7. Latest consensus state: current_term: 1 leader_uuid: "2cb2208698064486a07fb3122168e9f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.705572 30301 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:19:38.711793 30334 mvcc.cc:205] Tried to move back new op lower bound from 7333743733367517184 to 7333743733188419584. Current Snapshot: MvccSnapshot[applied={T|T < 7333743733367517184}]
I20260926 23:19:38.720847 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cb2208698064486a07fb3122168e9f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.721405 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:38.724936 30331 catalog_manager.cc:1347] Generated new cluster ID: ae80a25c3c0444f3b196df6d4fe34a5b
I20260926 23:19:38.725050 30331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:19:38.726199 30301 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cb2208698064486a07fb3122168e9f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.726461 30301 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:19:38.742247 30287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cb2208698064486a07fb3122168e9f7. Latest consensus state: current_term: 1 leader_uuid: "2cb2208698064486a07fb3122168e9f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.742619 30287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:19:38.757529 30315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cb2208698064486a07fb3122168e9f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.757815 30315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:38.759992 30333 mvcc.cc:205] Tried to move back new op lower bound from 7333743733367517184 to 7333743733188419584. Current Snapshot: MvccSnapshot[applied={T|T < 7333743733367517184}]
I20260926 23:19:38.788590 30287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cb2208698064486a07fb3122168e9f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ffda901b4a450baace2ec0266c45ce" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35495 } } peers { permanent_uuid: "2cb2208698064486a07fb3122168e9f7" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 35239 } } peers { permanent_uuid: "774a6300796b41bfbcc76bb201ffafd0" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 34861 } } }
I20260926 23:19:38.788897 30287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:19:38.826997 30331 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:19:38.829216 30331 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:19:38.852807 30331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cb2208698064486a07fb3122168e9f7: Generated new TSK 0
I20260926 23:19:38.854048 30331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:19:39.388386 30298 catalog_manager.cc:1259] Loaded cluster ID: ae80a25c3c0444f3b196df6d4fe34a5b
I20260926 23:19:39.388479 30298 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0: loading cluster ID for follower catalog manager: success
I20260926 23:19:39.390753 30298 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0: acquiring CA information for follower catalog manager: success
I20260926 23:19:39.392508 30298 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 774a6300796b41bfbcc76bb201ffafd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:19:39.450033 30312 catalog_manager.cc:1259] Loaded cluster ID: ae80a25c3c0444f3b196df6d4fe34a5b
I20260926 23:19:39.450186 30312 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce: loading cluster ID for follower catalog manager: success
I20260926 23:19:39.452673 30312 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce: acquiring CA information for follower catalog manager: success
I20260926 23:19:39.454447 30312 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1ffda901b4a450baace2ec0266c45ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:19:39.494725 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 30059
I20260926 23:19:39.496727 30121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:39.606688 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 30059
I20260926 23:19:39.627605 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 30133
I20260926 23:19:39.628803 30195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:39.736676 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 30133
I20260926 23:19:39.761605 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 30207
I20260926 23:19:39.762679 30269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:39.870924 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 30207
2026-09-26T23:19:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4416 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:19:39 dist-test-slave-vcv1 krb5kdc[30349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:19:39 dist-test-slave-vcv1 krb5kdc[30349](info): set up 2 sockets
Sep 26 23:19:39 dist-test-slave-vcv1 krb5kdc[30349](info): commencing operation
krb5kdc: starting...
W20260926 23:19:40.985664 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:19:41 dist-test-slave-vcv1 krb5kdc[30349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:19:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:19:41.228561 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:38473
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:38473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:41.559628 30362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:41.559942 30362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:41.560173 30362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:41.567626 30362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:19:41.567751 30362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:41.567838 30362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:41.567943 30362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:19:41.568022 30362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:19:41.578413 30362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:38473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:38473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:41.579799 30362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:41.581167 30362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:41.593281 30367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:41.593349 30368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:41.593380 30370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:41.594544 30362 server_base.cc:1034] running on GCE node
I20260926 23:19:41.595348 30362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:41.596897 30362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:41.598105 30362 hybrid_clock.cc:648] HybridClock initialized: now 1790464781598059 us; error 42 us; skew 500 ppm
I20260926 23:19:41.598465 30362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:41 dist-test-slave-vcv1 krb5kdc[30349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464781, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:41.601675 30362 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:19:41.603178 30362 webserver.cc:460] Webserver started at http://127.16.62.126:34723/ using document root <none> and password file <none>
I20260926 23:19:41.603922 30362 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:41.604051 30362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:41.604424 30362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:19:41.607806 30362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "4b1aaa62bf574dada4d2b7672417b9f7"
format_stamp: "Formatted at 2026-09-26 23:19:41 on dist-test-slave-vcv1"
server_key: "7c1e1d37ba06838fb9459bb9e8a1aa96"
server_key_iv: "188a76ee6226a234273f708b5d53692f"
server_key_version: "encryptionkey@0"
I20260926 23:19:41.608690 30362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "4b1aaa62bf574dada4d2b7672417b9f7"
format_stamp: "Formatted at 2026-09-26 23:19:41 on dist-test-slave-vcv1"
server_key: "7c1e1d37ba06838fb9459bb9e8a1aa96"
server_key_iv: "188a76ee6226a234273f708b5d53692f"
server_key_version: "encryptionkey@0"
I20260926 23:19:41.614342 30362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:19:41.618336 30377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:41.619786 30362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:19:41.620013 30362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "4b1aaa62bf574dada4d2b7672417b9f7"
format_stamp: "Formatted at 2026-09-26 23:19:41 on dist-test-slave-vcv1"
server_key: "7c1e1d37ba06838fb9459bb9e8a1aa96"
server_key_iv: "188a76ee6226a234273f708b5d53692f"
server_key_version: "encryptionkey@0"
I20260926 23:19:41.620268 30362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:41.637895 30362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:41.642144 30362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:41.642448 30362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:41.661751 30362 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:38473
I20260926 23:19:41.661770 30428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:38473 every 8 connection(s)
I20260926 23:19:41.663143 30362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:19:41.666781 30429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:41.672557 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 30362
I20260926 23:19:41.672772 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:19:41.673435 16633 external_mini_cluster.cc:1428] Setting key 5634371d902ca9a5936fb193c28b80bc
I20260926 23:19:41.677683 30429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7: Bootstrap starting.
I20260926 23:19:41.681445 30429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:41.682637 30429 log.cc:824] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:41.685493 30429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7: No bootstrap required, opened a new log
Sep 26 23:19:41 dist-test-slave-vcv1 krb5kdc[30349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:41.693861 30429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1aaa62bf574dada4d2b7672417b9f7" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38473 } }
I20260926 23:19:41.694293 30429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:41.694404 30429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1aaa62bf574dada4d2b7672417b9f7, State: Initialized, Role: FOLLOWER
I20260926 23:19:41.695114 30429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1aaa62bf574dada4d2b7672417b9f7" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38473 } }
I20260926 23:19:41.695374 30429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:19:41.695535 30429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:19:41.695761 30429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:41.697476 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:41.675342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.16.62.126:38473)
0926 23:19:41.676234 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:41.676246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:41.676286 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:19:41.678026 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:41.678047 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:41.678410 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:41.678724 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:41.680615 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:41.682115 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:41.683863 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:41.684610 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:41.684812 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:41.689198 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:41.689207 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:41.689221 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:41.689258 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:41.692691 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:41.693477 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:41.693485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:41.693492 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:41.693662 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:41.694093 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:41.694106 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:41.694108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:41.694874 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:41.695219 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:41.695754 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 23:19:41.696306 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":398,"threads_started":1}
I20260926 23:19:41.698453 30429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1aaa62bf574dada4d2b7672417b9f7" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38473 } }
I20260926 23:19:41.698992 30429 leader_election.cc:302] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b1aaa62bf574dada4d2b7672417b9f7; no voters:
I20260926 23:19:41.699602 30429 leader_election.cc:288] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:19:41.699779 30434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:19:41.701572 30434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [term 1 LEADER]: Becoming Leader. State: Replica: 4b1aaa62bf574dada4d2b7672417b9f7, State: Running, Role: LEADER
I20260926 23:19:41.701915 30429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:19:41.702136 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:19:41.702148 30434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1aaa62bf574dada4d2b7672417b9f7" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38473 } }
I20260926 23:19:41.705467 30436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b1aaa62bf574dada4d2b7672417b9f7. Latest consensus state: current_term: 1 leader_uuid: "4b1aaa62bf574dada4d2b7672417b9f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1aaa62bf574dada4d2b7672417b9f7" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38473 } } }
I20260926 23:19:41.705840 30436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:41.706434 30435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b1aaa62bf574dada4d2b7672417b9f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1aaa62bf574dada4d2b7672417b9f7" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 38473 } } }
I20260926 23:19:41.706735 30435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:41.709353 30444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:19:41.713944 30444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:19:41.723193 30444 catalog_manager.cc:1347] Generated new cluster ID: 7fe2689d36be49c6a2a6bcb20257eebf
I20260926 23:19:41.723335 30444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:19:41.754596 30444 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:19:41.756008 30444 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:19:41.768836 30444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b1aaa62bf574dada4d2b7672417b9f7: Generated new TSK 0
I20260926 23:19:41.769735 30444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:19:41.816844 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 30362
I20260926 23:19:41.818048 30424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:41.928869 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 30362
2026-09-26T23:19:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2057 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:19:42 dist-test-slave-vcv1 krb5kdc[30457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:19:42 dist-test-slave-vcv1 krb5kdc[30457](info): set up 2 sockets
Sep 26 23:19:42 dist-test-slave-vcv1 krb5kdc[30457](info): commencing operation
krb5kdc: starting...
W20260926 23:19:43.037657 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:19:43 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:19:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:19:43.272677 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:37363
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:37363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:43.599702 30470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:43.599982 30470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:43.600193 30470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:43.607518 30470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:19:43.607630 30470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:43.607739 30470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:43.607839 30470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:19:43.607904 30470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:19:43.617806 30470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:37363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:37363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:43.619105 30470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:43.620496 30470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:43.632391 30476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:43.632431 30478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:43.632611 30475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:43.633337 30470 server_base.cc:1034] running on GCE node
I20260926 23:19:43.634166 30470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:43.635531 30470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:43.636756 30470 hybrid_clock.cc:648] HybridClock initialized: now 1790464783636723 us; error 50 us; skew 500 ppm
I20260926 23:19:43.637112 30470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:43 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464783, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:43.647598 30470 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:19:43.649143 30470 webserver.cc:460] Webserver started at http://127.16.62.126:33819/ using document root <none> and password file <none>
I20260926 23:19:43.649837 30470 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:43.649981 30470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:43.650292 30470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:19:43.653402 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "1577316473b446d2a7a76c1b7263b81c"
format_stamp: "Formatted at 2026-09-26 23:19:43 on dist-test-slave-vcv1"
server_key: "1312e3e16f4522f5ac8685e4d90f425a"
server_key_iv: "6ca3b15264759a07c141893cb8376ccf"
server_key_version: "encryptionkey@0"
I20260926 23:19:43.654189 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "1577316473b446d2a7a76c1b7263b81c"
format_stamp: "Formatted at 2026-09-26 23:19:43 on dist-test-slave-vcv1"
server_key: "1312e3e16f4522f5ac8685e4d90f425a"
server_key_iv: "6ca3b15264759a07c141893cb8376ccf"
server_key_version: "encryptionkey@0"
I20260926 23:19:43.659447 30470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 23:19:43.662972 30485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:43.664377 30470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:19:43.664556 30470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "1577316473b446d2a7a76c1b7263b81c"
format_stamp: "Formatted at 2026-09-26 23:19:43 on dist-test-slave-vcv1"
server_key: "1312e3e16f4522f5ac8685e4d90f425a"
server_key_iv: "6ca3b15264759a07c141893cb8376ccf"
server_key_version: "encryptionkey@0"
I20260926 23:19:43.664749 30470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:43.683221 30470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:43.687057 30470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:43.687424 30470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:43.705711 30470 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:37363
I20260926 23:19:43.705751 30536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:37363 every 8 connection(s)
I20260926 23:19:43.707082 30470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:19:43.710696 30537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:43.714177 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 30470
I20260926 23:19:43.714406 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:19:43.715097 16633 external_mini_cluster.cc:1428] Setting key 3938c9cb456f08df86acafcef3256870
I20260926 23:19:43.724481 30537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Bootstrap starting.
I20260926 23:19:43.727989 30537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:43.729030 30537 log.cc:824] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:19:43 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:43.732300 30537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: No bootstrap required, opened a new log
I20260926 23:19:43.739087 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:43.717090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.16.62.126:37363)
0926 23:19:43.717833 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:43.717848 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:43.717897 (+ 49us) server_negotiation.cc:407] Connection header received
0926 23:19:43.720222 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:43.720257 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:43.720767 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:43.721207 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:43.722443 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:43.724424 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:43.725806 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:43.726886 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:43.727077 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:43.731418 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:43.731426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:43.731440 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:43.731473 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:43.734770 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:43.735659 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:43.735666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:43.735672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:43.735855 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:43.736322 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:43.736326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:43.736328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:43.736860 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:43.737120 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:43.737638 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 23:19:43.738024 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":295,"threads_started":1}
I20260926 23:19:43.741760 30537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:43.742229 30537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:43.742344 30537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1577316473b446d2a7a76c1b7263b81c, State: Initialized, Role: FOLLOWER
I20260926 23:19:43.743095 30537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:43.743323 30537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:19:43.743475 30537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:19:43.743662 30537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:43.746698 30537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:43.747298 30537 leader_election.cc:302] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1577316473b446d2a7a76c1b7263b81c; no voters:
I20260926 23:19:43.748030 30537 leader_election.cc:288] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:19:43.748196 30542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:19:43.750348 30537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:19:43.751030 30542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 1 LEADER]: Becoming Leader. State: Replica: 1577316473b446d2a7a76c1b7263b81c, State: Running, Role: LEADER
I20260926 23:19:43.751682 30542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:43.754782 30544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1577316473b446d2a7a76c1b7263b81c. Latest consensus state: current_term: 1 leader_uuid: "1577316473b446d2a7a76c1b7263b81c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } } }
I20260926 23:19:43.755100 30544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:43.755962 30554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:19:43.755760 30543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1577316473b446d2a7a76c1b7263b81c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } } }
I20260926 23:19:43.756210 30543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:43.761628 30554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:19:43.773785 30554 catalog_manager.cc:1347] Generated new cluster ID: ad04a3227393495ca10c46ec272dd4fd
I20260926 23:19:43.773900 30554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:19:43.796604 30554 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:19:43.798455 30554 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:19:43.811075 30554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Generated new TSK 0
I20260926 23:19:43.812235 30554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:19:43.867238 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:37363
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:44.194224 30563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:44.194502 30563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:44.194595 30563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:44.201843 30563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:44.201948 30563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:44.202067 30563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:19:44.211521 30563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:37363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:44.212723 30563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:44.213944 30563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:44.226995 30571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:44.227077 30569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:44.227698 30568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:44.228538 30563 server_base.cc:1034] running on GCE node
I20260926 23:19:44.229202 30563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:44.230166 30563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:44.231400 30563 hybrid_clock.cc:648] HybridClock initialized: now 1790464784231355 us; error 36 us; skew 500 ppm
I20260926 23:19:44.231719 30563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:44 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464784, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:44.234982 30563 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:19:44.236600 30563 webserver.cc:460] Webserver started at http://127.16.62.65:44729/ using document root <none> and password file <none>
I20260926 23:19:44.237293 30563 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:44.237443 30563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:44.237746 30563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:19:44.240778 30563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "af571e43ea33442b911da61ee60015ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "6a165683996ae1c93a84708de8d2a67e"
server_key_iv: "f67108af21ed97032685c19bd55a299f"
server_key_version: "encryptionkey@0"
I20260926 23:19:44.241605 30563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "af571e43ea33442b911da61ee60015ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "6a165683996ae1c93a84708de8d2a67e"
server_key_iv: "f67108af21ed97032685c19bd55a299f"
server_key_version: "encryptionkey@0"
I20260926 23:19:44.246874 30563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:19:44.250469 30578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:44.252019 30563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 23:19:44.252295 30563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "af571e43ea33442b911da61ee60015ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "6a165683996ae1c93a84708de8d2a67e"
server_key_iv: "f67108af21ed97032685c19bd55a299f"
server_key_version: "encryptionkey@0"
I20260926 23:19:44.252518 30563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:44.273661 30563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:44.278261 30563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:44.278586 30563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:44.279618 30563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:44.281095 30563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:44.281214 30563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:44.281383 30563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:44.281463 30563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:44.317551 30563 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:37859
I20260926 23:19:44.317593 30690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:37859 every 8 connection(s)
I20260926 23:19:44.318922 30563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:19:44.321578 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 30563
I20260926 23:19:44.321861 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:19:44.322633 16633 external_mini_cluster.cc:1428] Setting key 403c7ca9b340cbe310ae5aa7c2f88c54
Sep 26 23:19:44 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464784, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:44.343088 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:44.321590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:39643 (local address 127.16.62.126:37363)
0926 23:19:44.322067 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:44.322073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:44.323264 (+ 1191us) server_negotiation.cc:407] Connection header received
0926 23:19:44.325726 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:44.325731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:44.325792 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:44.325874 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:44.328080 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:44.328849 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:44.331492 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:44.331882 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:44.331974 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:44.336116 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:44.336132 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:44.336154 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:44.336200 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:44.339752 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:44.340578 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:44.340586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:44.340593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:44.340745 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:44.341149 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:44.341156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:44.341162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:44.341529 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:44.341684 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:44.342357 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 23:19:44.342610 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":237,"threads_started":1}
I20260926 23:19:44.343700 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:44.322156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:39643)
0926 23:19:44.323068 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:44.323122 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:44.325407 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:44.326171 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:44.326185 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:44.327020 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:44.327878 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:44.327910 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:44.329027 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:44.329037 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:44.331337 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:44.331357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:44.332125 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:44.332162 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:44.332508 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:44.333292 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:44.333331 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:44.335895 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:44.339915 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:44.339922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:44.339935 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:44.340453 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:44.340852 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:44.340855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:44.340857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:44.341022 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:44.341645 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:44.341650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:44.341968 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 23:19:44.342188 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 23:19:44.342567 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":318,"threads_started":1}
I20260926 23:19:44.345384 30691 heartbeater.cc:347] Connected to a master server at 127.16.62.126:37363
I20260926 23:19:44.345738 30691 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:44.346711 30691 heartbeater.cc:511] Master 127.16.62.126:37363 requested a full tablet report, sending...
I20260926 23:19:44.348994 30502 ts_manager.cc:194] Registered new tserver with Master: af571e43ea33442b911da61ee60015ee (127.16.62.65:37859)
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
I20260926 23:19:44.350592 30502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:39643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:19:44.380548 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:37363
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:44.703922 30697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:44.704259 30697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:44.704380 30697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:44.711731 30697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:44.711838 30697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:44.712030 30697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:19:44.721649 30697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:37363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:44.722889 30697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:44.724261 30697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:44.737305 30705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:44.737341 30703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:44.737306 30702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:44.738128 30697 server_base.cc:1034] running on GCE node
I20260926 23:19:44.738734 30697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:44.739724 30697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:44.740953 30697 hybrid_clock.cc:648] HybridClock initialized: now 1790464784740901 us; error 39 us; skew 500 ppm
I20260926 23:19:44.741287 30697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:44 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464784, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:44.744426 30697 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:19:44.745791 30697 webserver.cc:460] Webserver started at http://127.16.62.66:33119/ using document root <none> and password file <none>
I20260926 23:19:44.746445 30697 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:44.746544 30697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:44.746800 30697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:19:44.749754 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "f79be2c930ef409e9f9fff03c0c521ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "ff545c82ada71149ccc594aea3e19eac"
server_key_iv: "47674d2cfee15b95ae851238dad2473e"
server_key_version: "encryptionkey@0"
I20260926 23:19:44.750464 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "f79be2c930ef409e9f9fff03c0c521ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "ff545c82ada71149ccc594aea3e19eac"
server_key_iv: "47674d2cfee15b95ae851238dad2473e"
server_key_version: "encryptionkey@0"
I20260926 23:19:44.755829 30697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 23:19:44.759258 30712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:44.760739 30697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 23:19:44.760864 30697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "f79be2c930ef409e9f9fff03c0c521ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "ff545c82ada71149ccc594aea3e19eac"
server_key_iv: "47674d2cfee15b95ae851238dad2473e"
server_key_version: "encryptionkey@0"
I20260926 23:19:44.761020 30697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:44.781065 30697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:44.784953 30697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:44.785185 30697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:44.786069 30697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:44.787292 30697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:44.787364 30697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:44.787442 30697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:44.787494 30697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:44.819927 30697 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:37391
I20260926 23:19:44.820010 30824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:37391 every 8 connection(s)
I20260926 23:19:44.821278 30697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:19:44.821694 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 30697
I20260926 23:19:44.821938 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:19:44.822614 16633 external_mini_cluster.cc:1428] Setting key d57e76a8878d3b63e6efbe8489cbb486
Sep 26 23:19:44 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464784, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:44.842401 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:44.824085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:45599 (local address 127.16.62.126:37363)
0926 23:19:44.824283 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:44.824287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:44.825695 (+ 1408us) server_negotiation.cc:407] Connection header received
0926 23:19:44.827498 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:44.827507 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:44.827599 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:44.827717 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:44.829342 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:44.830526 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:44.832062 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:44.832582 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:44.832721 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:44.836388 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:44.836393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:44.836396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:44.836428 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:44.838989 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:44.839994 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:44.840005 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:44.840007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:44.840182 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:44.840638 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:44.840643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:44.840646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:44.840925 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:44.841089 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:44.841790 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 23:19:44.842031 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:19:44.843405 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:44.824727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:45599)
0926 23:19:44.825522 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:44.825573 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:44.827236 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:44.827907 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:44.827914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:44.828527 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:44.829174 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:44.829192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:44.830681 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:44.830685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:44.831915 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:44.831926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:44.832818 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:44.832824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:44.833053 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:44.833731 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:44.833757 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:44.836111 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:44.839138 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:44.839147 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:44.839164 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:44.839845 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:44.840307 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:44.840312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:44.840316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:44.840512 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:44.841057 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:44.841065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:44.841417 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 23:19:44.841685 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 23:19:44.842234 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":348,"threads_started":1}
I20260926 23:19:44.844944 30825 heartbeater.cc:347] Connected to a master server at 127.16.62.126:37363
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
I20260926 23:19:44.845255 30825 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:44.845995 30825 heartbeater.cc:511] Master 127.16.62.126:37363 requested a full tablet report, sending...
I20260926 23:19:44.847528 30502 ts_manager.cc:194] Registered new tserver with Master: f79be2c930ef409e9f9fff03c0c521ee (127.16.62.66:37391)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
I20260926 23:19:44.848730 30502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:45599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:19:44.876056 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:37363
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:45.194670 30830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:45.194928 30830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:45.195021 30830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:45.202086 30830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:45.202185 30830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:45.202294 30830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:19:45.212250 30830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:37363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:45.213414 30830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:45.214646 30830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:45.227401 30836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:45.227442 30835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:45.227490 30838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:45.228228 30830 server_base.cc:1034] running on GCE node
I20260926 23:19:45.228757 30830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:45.229585 30830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:45.230816 30830 hybrid_clock.cc:648] HybridClock initialized: now 1790464785230769 us; error 42 us; skew 500 ppm
I20260926 23:19:45.231117 30830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:45 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464785, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:45.234332 30830 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:19:45.235780 30830 webserver.cc:460] Webserver started at http://127.16.62.67:39343/ using document root <none> and password file <none>
I20260926 23:19:45.236509 30830 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:45.236657 30830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:45.236958 30830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:19:45.239787 30830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "13e39727907a4bb9becc5838a2172e3e"
format_stamp: "Formatted at 2026-09-26 23:19:45 on dist-test-slave-vcv1"
server_key: "50cf1f0f22920d18416611a21c1f9a45"
server_key_iv: "a15e558fe42a351b8885a24f10c97786"
server_key_version: "encryptionkey@0"
I20260926 23:19:45.240576 30830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "13e39727907a4bb9becc5838a2172e3e"
format_stamp: "Formatted at 2026-09-26 23:19:45 on dist-test-slave-vcv1"
server_key: "50cf1f0f22920d18416611a21c1f9a45"
server_key_iv: "a15e558fe42a351b8885a24f10c97786"
server_key_version: "encryptionkey@0"
I20260926 23:19:45.245481 30830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:19:45.248898 30845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:45.250165 30830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:19:45.250357 30830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "13e39727907a4bb9becc5838a2172e3e"
format_stamp: "Formatted at 2026-09-26 23:19:45 on dist-test-slave-vcv1"
server_key: "50cf1f0f22920d18416611a21c1f9a45"
server_key_iv: "a15e558fe42a351b8885a24f10c97786"
server_key_version: "encryptionkey@0"
I20260926 23:19:45.250576 30830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:45.272822 30830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:45.277179 30830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:45.277462 30830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:45.278414 30830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:45.279807 30830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:45.279911 30830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:45.280077 30830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:45.280167 30830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:45.312630 30830 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:32949
I20260926 23:19:45.312682 30957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:32949 every 8 connection(s)
I20260926 23:19:45.313935 30830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:19:45.316866 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 30830
I20260926 23:19:45.317052 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:19:45.317716 16633 external_mini_cluster.cc:1428] Setting key 7ae5352508b827326b4c3b883635b06f
Sep 26 23:19:45 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464785, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:45.334818 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.316477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:60817 (local address 127.16.62.126:37363)
0926 23:19:45.316686 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:45.316692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:45.318166 (+ 1474us) server_negotiation.cc:407] Connection header received
0926 23:19:45.320827 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:45.320832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:45.320917 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:45.321031 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:45.322503 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.323273 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.324903 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.325322 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.325458 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.328912 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:45.328918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:45.328923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:45.328966 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:45.331679 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:45.332545 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:45.332550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:45.332554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:45.332694 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:45.333118 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:45.333130 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:45.333134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:45.333398 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:45.333552 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:45.334246 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 23:19:45.334464 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:19:45.335749 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.317079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:60817)
0926 23:19:45.317994 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:45.318040 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:45.320462 (+ 2422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:45.321274 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:45.321283 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:45.321798 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:45.322334 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.322352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.323426 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.323432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.324759 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.324772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.325545 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.325554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.325795 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.326488 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:45.326511 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:45.328705 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:45.331821 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:45.331829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:45.331846 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:45.332417 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:45.332804 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:45.332810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:45.332814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:45.332998 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:45.333525 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:45.333533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:45.333869 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 23:19:45.334137 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 23:19:45.334657 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":334,"threads_started":1}
I20260926 23:19:45.337309 30958 heartbeater.cc:347] Connected to a master server at 127.16.62.126:37363
I20260926 23:19:45.337688 30958 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:45.338397 30958 heartbeater.cc:511] Master 127.16.62.126:37363 requested a full tablet report, sending...
I20260926 23:19:45.339751 30502 ts_manager.cc:194] Registered new tserver with Master: 13e39727907a4bb9becc5838a2172e3e (127.16.62.67:32949)
I20260926 23:19:45.340567 30502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:60817
I20260926 23:19:45.345366 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:19:45.353791 30691 heartbeater.cc:503] Master 127.16.62.126:37363 was elected leader, sending a full tablet report...
I20260926 23:19:45.362031 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.349688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.16.62.126:37363)
0926 23:19:45.349906 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:45.349912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:45.350137 (+ 225us) server_negotiation.cc:407] Connection header received
0926 23:19:45.350263 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:45.350266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:45.350324 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:45.350403 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:45.351813 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.352596 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.354061 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.354748 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.354887 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.357367 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:45.357372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:45.357376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:45.357405 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:45.359805 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:45.360485 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:45.360489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:45.360491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:45.360607 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:45.360916 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:45.360919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:45.360921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:45.361219 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:45.361397 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:45.361656 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 23:19:45.361814 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:19:45.365893 30502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:19:45.373772 30502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:19:45.436873 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.385619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.0.0.1:44570)
0926 23:19:45.424068 (+ 38449us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:45.424094 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:45.424272 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:45.429304 (+ 5032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:45.429309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:45.429343 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:45.430385 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.430396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.432211 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.432216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.433775 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.433787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.435736 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.435743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.435991 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.436055 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 23:19:45.436236 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 23:19:45.436469 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38253,"thread_start_us":38110,"threads_started":1}
I20260926 23:19:45.437330 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.385679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.16.62.66:37391)
0926 23:19:45.428311 (+ 42632us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:45.428322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:45.428354 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:19:45.428460 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:45.428468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:45.428720 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:45.428919 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:45.429834 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.431515 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.434614 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.436074 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.436574 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.436767 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:45.436834 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 23:19:45.436979 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42461,"thread_start_us":42335,"threads_started":1}
I20260926 23:19:45.437352 30973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.385457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:37391 (local address 127.0.0.1:44922)
0926 23:19:45.421163 (+ 35706us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:45.421226 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:45.421459 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:45.428971 (+ 7512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:45.428979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:45.429022 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:45.429605 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.429616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.431682 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.431687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.434455 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.434464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.436397 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.436403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.436667 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.436699 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:19:45.436783 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 23:19:45.436911 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35390,"mutex_wait_us":321,"spinlock_wait_cycles":896,"thread_start_us":41803,"threads_started":1}
I20260926 23:19:45.437577 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.386879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.16.62.67:32949)
0926 23:19:45.419616 (+ 32737us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:45.419630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:45.427786 (+ 8156us) server_negotiation.cc:407] Connection header received
0926 23:19:45.427983 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:45.427997 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:45.428404 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:45.428620 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:45.429852 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.431497 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.434607 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.436074 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.436241 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.436477 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:45.436817 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 23:19:45.437193 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32460,"thread_start_us":32779,"threads_started":1}
I20260926 23:19:45.438027 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.386215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.16.62.65:37859)
0926 23:19:45.428759 (+ 42544us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:45.428769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:45.428794 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:19:45.428900 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:45.428907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:45.429101 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:45.429308 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:45.430565 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.431655 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.434203 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.435499 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.435660 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.435931 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:45.437510 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0926 23:19:45.437696 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42366,"thread_start_us":42211,"threads_started":1}
I20260926 23:19:45.438377 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.386109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:32949 (local address 127.0.0.1:53164)
0926 23:19:45.427597 (+ 41488us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:45.427622 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:45.427783 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:45.428971 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:45.428979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:45.429028 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:45.429603 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.429616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.431685 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.431690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.434425 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.434447 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.436360 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.436370 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.436606 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.436640 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 23:19:45.437837 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0926 23:19:45.437969 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41307,"mutex_wait_us":191,"spinlock_wait_cycles":4352,"thread_start_us":35554,"threads_started":1}
I20260926 23:19:45.441645 30893 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa052a20cd047b59abc4c88e86b5b89 (DEFAULT_TABLE table=test-table [id=7a4c0264eb124f36a7496faf58419c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:19:45.441677 30626 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa052a20cd047b59abc4c88e86b5b89 (DEFAULT_TABLE table=test-table [id=7a4c0264eb124f36a7496faf58419c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:19:45.441670 30760 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa052a20cd047b59abc4c88e86b5b89 (DEFAULT_TABLE table=test-table [id=7a4c0264eb124f36a7496faf58419c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:19:45.442917 30893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa052a20cd047b59abc4c88e86b5b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:45.442976 30626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa052a20cd047b59abc4c88e86b5b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:45.442970 30760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa052a20cd047b59abc4c88e86b5b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:45.455320 30976 tablet_bootstrap.cc:492] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: Bootstrap starting.
I20260926 23:19:45.458947 30976 tablet_bootstrap.cc:654] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:45.459792 30977 tablet_bootstrap.cc:492] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: Bootstrap starting.
I20260926 23:19:45.460297 30976 log.cc:824] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:45.463270 30978 tablet_bootstrap.cc:492] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: Bootstrap starting.
I20260926 23:19:45.463953 30977 tablet_bootstrap.cc:654] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:45.465517 30977 log.cc:824] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:45.466693 30978 tablet_bootstrap.cc:654] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:45.467839 30978 log.cc:824] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:45.468608 30977 tablet_bootstrap.cc:492] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: No bootstrap required, opened a new log
I20260926 23:19:45.468894 30977 ts_tablet_manager.cc:1397] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260926 23:19:45.469060 30976 tablet_bootstrap.cc:492] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: No bootstrap required, opened a new log
I20260926 23:19:45.469353 30976 ts_tablet_manager.cc:1397] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260926 23:19:45.470969 30978 tablet_bootstrap.cc:492] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: No bootstrap required, opened a new log
I20260926 23:19:45.471212 30978 ts_tablet_manager.cc:1397] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:19:45.476624 30977 raft_consensus.cc:348] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } }
I20260926 23:19:45.476981 30977 raft_consensus.cc:374] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:45.477090 30977 raft_consensus.cc:729] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af571e43ea33442b911da61ee60015ee, State: Initialized, Role: FOLLOWER
I20260926 23:19:45.477700 30977 consensus_queue.cc:260] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } }
I20260926 23:19:45.479394 30977 ts_tablet_manager.cc:1428] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260926 23:19:45.480309 30976 raft_consensus.cc:348] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } }
I20260926 23:19:45.480710 30976 raft_consensus.cc:374] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:45.480814 30976 raft_consensus.cc:729] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e39727907a4bb9becc5838a2172e3e, State: Initialized, Role: FOLLOWER
I20260926 23:19:45.481385 30976 consensus_queue.cc:260] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } }
I20260926 23:19:45.481709 30978 raft_consensus.cc:348] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } }
I20260926 23:19:45.482196 30978 raft_consensus.cc:374] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:45.482329 30978 raft_consensus.cc:729] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79be2c930ef409e9f9fff03c0c521ee, State: Initialized, Role: FOLLOWER
I20260926 23:19:45.482481 30958 heartbeater.cc:503] Master 127.16.62.126:37363 was elected leader, sending a full tablet report...
I20260926 23:19:45.483142 30976 ts_tablet_manager.cc:1428] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260926 23:19:45.483035 30978 consensus_queue.cc:260] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } }
I20260926 23:19:45.485374 30825 heartbeater.cc:503] Master 127.16.62.126:37363 was elected leader, sending a full tablet report...
I20260926 23:19:45.485759 30978 ts_tablet_manager.cc:1428] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
W20260926 23:19:45.565560 30959 tablet.cc:2369] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:19:45.571310 30692 tablet.cc:2369] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:19:45.573585 30826 tablet.cc:2369] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:19:45.739001 30983 raft_consensus.cc:482] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:19:45.739496 30983 raft_consensus.cc:504] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } }
I20260926 23:19:45.741921 30983 leader_election.cc:288] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f79be2c930ef409e9f9fff03c0c521ee (127.16.62.66:37391), af571e43ea33442b911da61ee60015ee (127.16.62.65:37859)
I20260926 23:19:45.749646 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.742201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:37391 (local address 127.16.62.67:42453)
0926 23:19:45.742499 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:45.742614 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:45.742826 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:45.743332 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:45.743337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:45.743406 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:45.743839 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.743855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.745544 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.745550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.747639 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.747653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.749100 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.749104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.749232 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.749256 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 23:19:45.749330 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 23:19:45.749426 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 23:19:45.749933 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.742604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:42453 (local address 127.16.62.66:37391)
0926 23:19:45.742850 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:45.742856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:45.742882 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:19:45.743077 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:45.743083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:45.743174 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:45.743271 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:45.744062 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.745373 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.747815 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.748925 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.749111 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.749204 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:45.749389 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 23:19:45.749661 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 23:19:45.751101 30780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa052a20cd047b59abc4c88e86b5b89" candidate_uuid: "13e39727907a4bb9becc5838a2172e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79be2c930ef409e9f9fff03c0c521ee" is_pre_election: true
I20260926 23:19:45.751341 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.742549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:58339 (local address 127.16.62.65:37859)
0926 23:19:45.742807 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:45.742815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:45.744682 (+ 1867us) server_negotiation.cc:407] Connection header received
0926 23:19:45.744853 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:45.744859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:45.744942 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:45.745089 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:45.745619 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.746719 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.748893 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.750334 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.750505 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.750577 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:45.751053 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 23:19:45.751170 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:19:45.751693 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.743169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.16.62.67:58339)
0926 23:19:45.744425 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:45.744558 (+ 133us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:45.744708 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:45.745102 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:45.745106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:45.745131 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:45.745435 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.745443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.746886 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.746891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.748727 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:45.748739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.750665 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.750674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:45.750859 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.750890 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:19:45.751007 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 23:19:45.751182 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":417,"threads_started":1}
I20260926 23:19:45.751761 30780 raft_consensus.cc:2454] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e39727907a4bb9becc5838a2172e3e in term 0.
I20260926 23:19:45.752489 30849 leader_election.cc:302] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e39727907a4bb9becc5838a2172e3e, f79be2c930ef409e9f9fff03c0c521ee; no voters:
I20260926 23:19:45.752920 30983 raft_consensus.cc:2781] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:19:45.753051 30983 raft_consensus.cc:482] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:19:45.753170 30983 raft_consensus.cc:3037] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:45.753082 30646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa052a20cd047b59abc4c88e86b5b89" candidate_uuid: "13e39727907a4bb9becc5838a2172e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af571e43ea33442b911da61ee60015ee" is_pre_election: true
I20260926 23:19:45.753639 30646 raft_consensus.cc:2454] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e39727907a4bb9becc5838a2172e3e in term 0.
I20260926 23:19:45.756101 30983 raft_consensus.cc:504] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } }
I20260926 23:19:45.756878 30983 leader_election.cc:288] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [CANDIDATE]: Term 1 election: Requested vote from peers f79be2c930ef409e9f9fff03c0c521ee (127.16.62.66:37391), af571e43ea33442b911da61ee60015ee (127.16.62.65:37859)
I20260926 23:19:45.757362 30780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa052a20cd047b59abc4c88e86b5b89" candidate_uuid: "13e39727907a4bb9becc5838a2172e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79be2c930ef409e9f9fff03c0c521ee"
I20260926 23:19:45.757581 30780 raft_consensus.cc:3037] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:45.757495 30646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa052a20cd047b59abc4c88e86b5b89" candidate_uuid: "13e39727907a4bb9becc5838a2172e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af571e43ea33442b911da61ee60015ee"
I20260926 23:19:45.757838 30646 raft_consensus.cc:3037] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:45.760031 30646 raft_consensus.cc:2454] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e39727907a4bb9becc5838a2172e3e in term 1.
I20260926 23:19:45.760315 30780 raft_consensus.cc:2454] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e39727907a4bb9becc5838a2172e3e in term 1.
I20260926 23:19:45.760569 30846 leader_election.cc:302] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e39727907a4bb9becc5838a2172e3e, af571e43ea33442b911da61ee60015ee; no voters:
I20260926 23:19:45.760918 30983 raft_consensus.cc:2781] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:19:45.761400 30983 raft_consensus.cc:686] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 1 LEADER]: Becoming Leader. State: Replica: 13e39727907a4bb9becc5838a2172e3e, State: Running, Role: LEADER
I20260926 23:19:45.761929 30983 consensus_queue.cc:237] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } }
I20260926 23:19:45.766956 30500 catalog_manager.cc:5495] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e39727907a4bb9becc5838a2172e3e (127.16.62.67). New cstate: current_term: 1 leader_uuid: "13e39727907a4bb9becc5838a2172e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } health_report { overall_health: HEALTHY } } }
I20260926 23:19:45.801198 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.795758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.16.62.67:32949)
0926 23:19:45.795981 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:45.795985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:45.796026 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:19:45.796194 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:45.796197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:45.796298 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:45.796431 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:19:45.796761 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.797490 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.799164 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.799559 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.799651 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.800129 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:19:45.800717 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:19:45.800824 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:45.800930 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 23:19:45.800997 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 23:19:45.809346 30780 raft_consensus.cc:1261] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [term 1 FOLLOWER]: Refusing update from remote peer 13e39727907a4bb9becc5838a2172e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:19:45.809427 30646 raft_consensus.cc:1261] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [term 1 FOLLOWER]: Refusing update from remote peer 13e39727907a4bb9becc5838a2172e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:19:45.810380 30986 consensus_queue.cc:1035] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:19:45.810729 30983 consensus_queue.cc:1035] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:19:45.824658 30989 mvcc.cc:205] Tried to move back new op lower bound from 7333743762664710144 to 7333743762486239232. Current Snapshot: MvccSnapshot[applied={T|T < 7333743762664710144}]
I20260926 23:19:45.845485 30502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:19:45.845803 30502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:19:45.849956 30502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Sending DeleteTablet for 3 replicas of tablet 5aa052a20cd047b59abc4c88e86b5b89
I20260926 23:19:45.851253 30760 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa052a20cd047b59abc4c88e86b5b89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:19:45 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:44922
I20260926 23:19:45.851248 30626 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa052a20cd047b59abc4c88e86b5b89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:19:45 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:44570
I20260926 23:19:45.851711 30893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa052a20cd047b59abc4c88e86b5b89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:19:45 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:53164
I20260926 23:19:45.853065 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 30563
I20260926 23:19:45.864538 30998 tablet_replica.cc:330] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: stopping tablet replica
I20260926 23:19:45.865322 30998 raft_consensus.cc:2229] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:19:45.866050 30998 raft_consensus.cc:2258] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:19:45.869376 30999 tablet_replica.cc:330] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: stopping tablet replica
I20260926 23:19:45.869866 30999 raft_consensus.cc:2229] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:19:45.870419 30999 raft_consensus.cc:2258] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:19:45.870841 30998 ts_tablet_manager.cc:1905] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:19:45.874087 30999 ts_tablet_manager.cc:1905] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:19:45.878870 30998 ts_tablet_manager.cc:1918] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:45.879065 30998 log.cc:1196] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/5aa052a20cd047b59abc4c88e86b5b89
I20260926 23:19:45.879613 30998 ts_tablet_manager.cc:1939] T 5aa052a20cd047b59abc4c88e86b5b89 P f79be2c930ef409e9f9fff03c0c521ee: Deleting consensus metadata
I20260926 23:19:45.880355 31000 tablet_replica.cc:330] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: stopping tablet replica
I20260926 23:19:45.881709 30489 catalog_manager.cc:4841] TS f79be2c930ef409e9f9fff03c0c521ee (127.16.62.66:37391): tablet 5aa052a20cd047b59abc4c88e86b5b89 (table test-table [id=7a4c0264eb124f36a7496faf58419c08]) successfully deleted
I20260926 23:19:45.882303 30999 ts_tablet_manager.cc:1918] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:45.882478 30999 log.cc:1196] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/5aa052a20cd047b59abc4c88e86b5b89
I20260926 23:19:45.882355 31000 raft_consensus.cc:2229] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:19:45.883076 30999 ts_tablet_manager.cc:1939] T 5aa052a20cd047b59abc4c88e86b5b89 P af571e43ea33442b911da61ee60015ee: Deleting consensus metadata
I20260926 23:19:45.883248 31000 raft_consensus.cc:2258] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:19:45.885375 30486 catalog_manager.cc:4841] TS af571e43ea33442b911da61ee60015ee (127.16.62.65:37859): tablet 5aa052a20cd047b59abc4c88e86b5b89 (table test-table [id=7a4c0264eb124f36a7496faf58419c08]) successfully deleted
I20260926 23:19:45.885457 31000 ts_tablet_manager.cc:1905] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 23:19:45 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:19:45.892336 31000 ts_tablet_manager.cc:1918] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:45.892521 31000 log.cc:1196] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/5aa052a20cd047b59abc4c88e86b5b89
I20260926 23:19:45.893101 31000 ts_tablet_manager.cc:1939] T 5aa052a20cd047b59abc4c88e86b5b89 P 13e39727907a4bb9becc5838a2172e3e: Deleting consensus metadata
I20260926 23:19:45.895032 30487 catalog_manager.cc:4841] TS 13e39727907a4bb9becc5838a2172e3e (127.16.62.67:32949): tablet 5aa052a20cd047b59abc4c88e86b5b89 (table test-table [id=7a4c0264eb124f36a7496faf58419c08]) successfully deleted
I20260926 23:19:45.898383 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:45.854577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.16.62.65:37859)
0926 23:19:45.854810 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:45.854819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:45.877048 (+ 22229us) server_negotiation.cc:407] Connection header received
0926 23:19:45.877558 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:45.877564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:45.877647 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:45.877728 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:45.880912 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.882021 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.884231 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:45.884861 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:45.884962 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:45.891058 (+ 6096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:45.891076 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:45.891093 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:45.891148 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:45.895210 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:45.896282 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:45.896288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:45.896290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:45.896422 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:45.896898 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:45.896901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:45.896903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:45.897263 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:45.897483 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:45.897912 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 23:19:45.898070 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:19:45.899055 30686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:46.009408 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 30563
I20260926 23:19:46.026844 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 30697
Sep 26 23:19:46 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:19:46.047049 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:46.027706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.16.62.66:37391)
0926 23:19:46.027866 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:46.027871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:46.027938 (+ 67us) server_negotiation.cc:407] Connection header received
0926 23:19:46.028356 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:46.028361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:46.028430 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:46.028518 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:46.030410 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:46.031340 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:46.033488 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:46.033901 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:46.034001 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:46.039555 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:46.039577 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:46.039599 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:46.039667 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:46.043680 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:46.044513 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:46.044524 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:46.044529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:46.044713 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:46.045124 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:46.045130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:46.045134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:46.045671 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:46.045984 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:46.046293 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 23:19:46.046545 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 23:19:46.047868 30820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:46.137291 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 30697
I20260926 23:19:46.162572 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 30830
Sep 26 23:19:46 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:19:46.182060 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:46.163529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.16.62.67:32949)
0926 23:19:46.163701 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:46.163706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:46.163723 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:19:46.163924 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:46.163928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:46.163990 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:46.164071 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:46.166034 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:46.166864 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:46.169115 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:46.169562 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:46.169703 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:46.175640 (+ 5937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:46.175654 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:46.175680 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:46.175730 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:46.178911 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:46.179909 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:46.179916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:46.179918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:46.180051 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:46.180553 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:46.180557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:46.180559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:46.180912 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:46.181129 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:46.181596 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 23:19:46.181765 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:19:46.182912 30953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:46.303948 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 30830
I20260926 23:19:46.323292 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 30470
I20260926 23:19:46.324553 30532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:46.436986 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 30470
I20260926 23:19:46.458479 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:37363
--webserver_interface=127.16.62.126
--webserver_port=33819
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:37363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:46.789006 31006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:46.789296 31006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:46.789384 31006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:46.796514 31006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:19:46.796612 31006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:46.796669 31006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:46.796725 31006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:19:46.796770 31006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:19:46.806393 31006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:37363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:37363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=33819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:46.807595 31006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:46.808859 31006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:46.831854 31012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:46.831792 31013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:46.831877 31015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:46.833076 31006 server_base.cc:1034] running on GCE node
I20260926 23:19:46.834002 31006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:46.835708 31006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:46.836982 31006 hybrid_clock.cc:648] HybridClock initialized: now 1790464786836915 us; error 85 us; skew 500 ppm
I20260926 23:19:46.837404 31006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:46 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464786, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:46.841073 31006 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:19:46.842767 31006 webserver.cc:460] Webserver started at http://127.16.62.126:33819/ using document root <none> and password file <none>
I20260926 23:19:46.843495 31006 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:46.843616 31006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:46.849875 31006 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:19:46.852792 31022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:46.854175 31006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 23:19:46.854315 31006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "1577316473b446d2a7a76c1b7263b81c"
format_stamp: "Formatted at 2026-09-26 23:19:43 on dist-test-slave-vcv1"
server_key: "1312e3e16f4522f5ac8685e4d90f425a"
server_key_iv: "6ca3b15264759a07c141893cb8376ccf"
server_key_version: "encryptionkey@0"
I20260926 23:19:46.855350 31006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:46.880710 31006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:46.885315 31006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:46.885543 31006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:46.904723 31006 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:37363
I20260926 23:19:46.904731 31073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:37363 every 8 connection(s)
I20260926 23:19:46.906265 31006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:19:46.910677 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 31006
I20260926 23:19:46.911696 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:37859
--local_ip_for_outbound_sockets=127.16.62.65
--tserver_master_addrs=127.16.62.126:37363
--webserver_port=44729
--webserver_interface=127.16.62.65
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:19:46.914798 31074 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:19:46.920289 31074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Bootstrap starting.
I20260926 23:19:46.940331 31074 log.cc:824] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:46.952528 31074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:19:46.953091 31074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Bootstrap complete.
I20260926 23:19:46.962788 31074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:46.963696 31074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1577316473b446d2a7a76c1b7263b81c, State: Initialized, Role: FOLLOWER
I20260926 23:19:46.964382 31074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:46.964656 31074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:19:46.964792 31074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:19:46.964973 31074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:19:46.968952 31074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:46.969518 31074 leader_election.cc:302] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1577316473b446d2a7a76c1b7263b81c; no voters:
I20260926 23:19:46.970233 31074 leader_election.cc:288] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:19:46.970366 31078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:19:46.971587 31078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 2 LEADER]: Becoming Leader. State: Replica: 1577316473b446d2a7a76c1b7263b81c, State: Running, Role: LEADER
I20260926 23:19:46.972111 31078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:46.972635 31074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:19:46.974701 31079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1577316473b446d2a7a76c1b7263b81c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } } }
I20260926 23:19:46.975139 31079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:46.978982 31080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1577316473b446d2a7a76c1b7263b81c. Latest consensus state: current_term: 2 leader_uuid: "1577316473b446d2a7a76c1b7263b81c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } } }
I20260926 23:19:46.979359 31080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:46.981210 31084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:19:46.990088 31084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:19:46.991503 31084 catalog_manager.cc:1259] Loaded cluster ID: ad04a3227393495ca10c46ec272dd4fd
I20260926 23:19:46.991605 31084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:19:46.994800 31084 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 23:19:47.012060 31084 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 23:19:47.021142 31084 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:19:47.022866 31084 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Loaded TSK: 0
I20260926 23:19:47.024406 31084 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:19:47.256551 31076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:47.256887 31076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:47.257061 31076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:47.264755 31076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:47.264915 31076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:47.265122 31076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:19:47.275187 31076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:37859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=44729
--tserver_master_addrs=127.16.62.126:37363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:47.276667 31076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:47.278028 31076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:47.292917 31101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:47.292896 31104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:47.293154 31102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:47.293989 31076 server_base.cc:1034] running on GCE node
I20260926 23:19:47.294629 31076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:47.295560 31076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:47.296780 31076 hybrid_clock.cc:648] HybridClock initialized: now 1790464787296731 us; error 58 us; skew 500 ppm
I20260926 23:19:47.297092 31076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:47 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464787, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:47.300203 31076 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:19:47.301687 31076 webserver.cc:460] Webserver started at http://127.16.62.65:44729/ using document root <none> and password file <none>
I20260926 23:19:47.302361 31076 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:47.302503 31076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:47.307976 31076 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:19:47.310709 31111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:47.312069 31076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:19:47.312263 31076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "af571e43ea33442b911da61ee60015ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "6a165683996ae1c93a84708de8d2a67e"
server_key_iv: "f67108af21ed97032685c19bd55a299f"
server_key_version: "encryptionkey@0"
I20260926 23:19:47.313217 31076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:47.337565 31076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:47.342672 31076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:47.342963 31076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:47.344012 31076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:47.345463 31076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:47.345607 31076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:47.345736 31076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:47.345824 31076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:47.379784 31076 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:37859
I20260926 23:19:47.379873 31223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:37859 every 8 connection(s)
I20260926 23:19:47.381309 31076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:19:47.387930 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 31076
I20260926 23:19:47.388832 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:37391
--local_ip_for_outbound_sockets=127.16.62.66
--tserver_master_addrs=127.16.62.126:37363
--webserver_port=33119
--webserver_interface=127.16.62.66
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:19:47 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464787, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:47.406383 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:47.384345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:38055 (local address 127.16.62.126:37363)
0926 23:19:47.384868 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:47.384879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:47.385633 (+ 754us) server_negotiation.cc:407] Connection header received
0926 23:19:47.387652 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:47.387664 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:47.387970 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:47.388339 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:47.390317 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:47.391535 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:47.394095 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:47.394741 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:47.394934 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:47.399019 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:47.399027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:47.399039 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:47.399073 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:47.401981 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:47.402710 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:47.402716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:47.402721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:47.402874 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:47.403301 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:47.403310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:47.403313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:47.403757 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:47.403965 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:47.404790 (+ 825us) server_negotiation.cc:299] Negotiation successful
0926 23:19:47.405150 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":220,"threads_started":1}
I20260926 23:19:47.406383 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:47.384617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:38055)
0926 23:19:47.385467 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:47.385517 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:47.387199 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:47.388543 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:47.388555 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:47.389297 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:47.390084 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:47.390112 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:47.391720 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:47.391728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:47.393957 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:47.393974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:47.394975 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:47.394983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:47.395255 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:47.395928 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:47.395950 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:47.398741 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:47.402131 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:47.402143 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:47.402156 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:47.402555 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:47.402990 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:47.402995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:47.402998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:47.403181 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:47.403901 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:47.403909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:47.404300 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 23:19:47.404555 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 23:19:47.405082 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":358,"threads_started":1}
I20260926 23:19:47.409415 31224 heartbeater.cc:347] Connected to a master server at 127.16.62.126:37363
I20260926 23:19:47.409722 31224 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:47.410411 31224 heartbeater.cc:511] Master 127.16.62.126:37363 requested a full tablet report, sending...
I20260926 23:19:47.412602 31039 ts_manager.cc:194] Registered new tserver with Master: af571e43ea33442b911da61ee60015ee (127.16.62.65:37859)
I20260926 23:19:47.414091 31039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:38055
W20260926 23:19:47.715230 31228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:47.715544 31228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:47.715742 31228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:47.723037 31228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:47.723205 31228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:47.723381 31228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:19:47.733152 31228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:37391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=33119
--tserver_master_addrs=127.16.62.126:37363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:47.734476 31228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:47.735808 31228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:47.751205 31234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:47.751328 31233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:47.752329 31228 server_base.cc:1034] running on GCE node
W20260926 23:19:47.752424 31236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:47.753180 31228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:47.754156 31228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:47.755396 31228 hybrid_clock.cc:648] HybridClock initialized: now 1790464787755361 us; error 47 us; skew 500 ppm
I20260926 23:19:47.755751 31228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:47 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464787, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:47.758895 31228 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:19:47.760483 31228 webserver.cc:460] Webserver started at http://127.16.62.66:33119/ using document root <none> and password file <none>
I20260926 23:19:47.761114 31228 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:47.761226 31228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:47.766422 31228 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 23:19:47.769004 31243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:47.770349 31228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:19:47.770511 31228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "f79be2c930ef409e9f9fff03c0c521ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "ff545c82ada71149ccc594aea3e19eac"
server_key_iv: "47674d2cfee15b95ae851238dad2473e"
server_key_version: "encryptionkey@0"
I20260926 23:19:47.771440 31228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:47.793901 31228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:47.798034 31228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:47.798316 31228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:47.799284 31228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:47.800643 31228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:47.800753 31228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:47.800880 31228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:47.800952 31228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:47.834553 31228 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:37391
I20260926 23:19:47.834640 31355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:37391 every 8 connection(s)
I20260926 23:19:47.836026 31228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:19:47.843259 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 31228
I20260926 23:19:47.843868 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:32949
--local_ip_for_outbound_sockets=127.16.62.67
--tserver_master_addrs=127.16.62.126:37363
--webserver_port=39343
--webserver_interface=127.16.62.67
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:19:47 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464787, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:47.856355 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:47.838550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:38085 (local address 127.16.62.126:37363)
0926 23:19:47.838761 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:47.838767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:47.839947 (+ 1180us) server_negotiation.cc:407] Connection header received
0926 23:19:47.841838 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:47.841843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:47.841909 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:47.841998 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:47.843491 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:47.844320 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:47.845851 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:47.846644 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:47.846777 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:47.850561 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:47.850566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:47.850570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:47.850603 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:47.853297 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:47.854011 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:47.854019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:47.854025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:47.854164 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:47.854590 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:47.854596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:47.854599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:47.854872 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:47.855027 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:47.855740 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 23:19:47.855971 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:19:47.857332 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:47.839091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:38085)
0926 23:19:47.839782 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:47.839825 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:47.841578 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:47.842200 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:47.842208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:47.842702 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:47.843243 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:47.843259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:47.844483 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:47.844493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:47.845700 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:47.845713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:47.847061 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:47.847069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:47.847379 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:47.848057 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:47.848084 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:47.850338 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:47.853433 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:47.853441 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:47.853458 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:47.853884 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:47.854282 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:47.854287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:47.854291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:47.854474 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:47.855001 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:47.855008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:47.855343 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 23:19:47.855619 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 23:19:47.856189 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":265,"threads_started":1}
I20260926 23:19:47.858846 31356 heartbeater.cc:347] Connected to a master server at 127.16.62.126:37363
I20260926 23:19:47.859210 31356 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:47.860081 31356 heartbeater.cc:511] Master 127.16.62.126:37363 requested a full tablet report, sending...
I20260926 23:19:47.861713 31039 ts_manager.cc:194] Registered new tserver with Master: f79be2c930ef409e9f9fff03c0c521ee (127.16.62.66:37391)
I20260926 23:19:47.863016 31039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:38085
W20260926 23:19:48.176903 31359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:48.177191 31359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:48.177287 31359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:48.184540 31359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:48.184667 31359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:48.184783 31359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:19:48.194447 31359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:32949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=39343
--tserver_master_addrs=127.16.62.126:37363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:48.195673 31359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:48.196954 31359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:48.209764 31364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:48.209724 31367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:48.209792 31365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:48.210937 31359 server_base.cc:1034] running on GCE node
I20260926 23:19:48.211635 31359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:48.212630 31359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:48.213824 31359 hybrid_clock.cc:648] HybridClock initialized: now 1790464788213763 us; error 51 us; skew 500 ppm
I20260926 23:19:48.214134 31359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:48 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464788, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:48.217404 31359 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:19:48.219022 31359 webserver.cc:460] Webserver started at http://127.16.62.67:39343/ using document root <none> and password file <none>
I20260926 23:19:48.219741 31359 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:48.219903 31359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:48.225311 31359 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 23:19:48.228366 31374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:48.229779 31359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:19:48.229965 31359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "13e39727907a4bb9becc5838a2172e3e"
format_stamp: "Formatted at 2026-09-26 23:19:45 on dist-test-slave-vcv1"
server_key: "50cf1f0f22920d18416611a21c1f9a45"
server_key_iv: "a15e558fe42a351b8885a24f10c97786"
server_key_version: "encryptionkey@0"
I20260926 23:19:48.230983 31359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:48.265373 31359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:48.269575 31359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:48.269876 31359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:48.270888 31359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:48.272224 31359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:48.272346 31359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:48.272519 31359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:48.272598 31359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:48.306655 31359 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:32949
I20260926 23:19:48.306701 31486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:32949 every 8 connection(s)
I20260926 23:19:48.308108 31359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:19:48.310227 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 31359
Sep 26 23:19:48 dist-test-slave-vcv1 krb5kdc[30457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464788, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:19:48.326299 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.311620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.16.62.126:37363)
0926 23:19:48.312117 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.312126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.312177 (+ 51us) server_negotiation.cc:407] Connection header received
0926 23:19:48.312261 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.312268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.312347 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.312469 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:48.314572 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.315316 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.316888 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.317338 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.317474 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.320406 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:48.320411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:48.320415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:48.320445 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:48.323190 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.323928 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.323934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.323936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.324060 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.324494 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.324497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.324499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.324901 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:48.325092 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.325529 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.325749 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":212,"threads_started":1}
I20260926 23:19:48.330062 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.310854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:35083 (local address 127.16.62.126:37363)
0926 23:19:48.311049 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.311054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.312849 (+ 1795us) server_negotiation.cc:407] Connection header received
0926 23:19:48.315269 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.315275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.315363 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.315467 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:48.317168 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.317976 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.319532 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.319971 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.320108 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.323953 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:48.323958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:48.323961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:48.324003 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:48.327269 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.327973 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.327983 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.327985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.328107 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.328658 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.328664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.328672 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.328909 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:48.329015 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.329672 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.329825 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:19:48.331054 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.311527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:35083)
0926 23:19:48.312669 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.312725 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:48.315001 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:48.315707 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:48.315715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:48.316314 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:19:48.316953 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.317018 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.318164 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.318169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.319387 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.319399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.320227 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.320236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.320561 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.321227 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:19:48.321259 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:19:48.323727 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:19:48.327401 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:48.327410 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:48.327427 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:48.327842 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:48.328265 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:19:48.328269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:19:48.328272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:19:48.328523 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:19:48.329018 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:19:48.329024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:19:48.329350 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 23:19:48.329569 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 23:19:48.329952 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":635,"threads_started":1}
I20260926 23:19:48.332528 31487 heartbeater.cc:347] Connected to a master server at 127.16.62.126:37363
I20260926 23:19:48.332830 31487 heartbeater.cc:464] Registering TS with master...
I20260926 23:19:48.333546 31487 heartbeater.cc:511] Master 127.16.62.126:37363 requested a full tablet report, sending...
I20260926 23:19:48.334896 31039 ts_manager.cc:194] Registered new tserver with Master: 13e39727907a4bb9becc5838a2172e3e (127.16.62.67:32949)
I20260926 23:19:48.335767 31039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:35083
I20260926 23:19:48.340972 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:19:48.347419 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.341871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.16.62.126:37363)
0926 23:19:48.342103 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.342109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.342436 (+ 327us) server_negotiation.cc:407] Connection header received
0926 23:19:48.342602 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.342605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.342741 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.342856 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:19:48.343242 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.343960 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.345398 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.345803 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.345896 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.346328 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:19:48.346822 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:19:48.346924 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.347076 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.347139 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:19:48.348409 31039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:19:48.355507 31039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:19:48.397596 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.368712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.16.62.65:37859)
0926 23:19:48.389388 (+ 20676us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.389396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.389419 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:19:48.389492 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.389497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.389671 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.389872 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:48.391298 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.392882 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.395423 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.396428 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.396578 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.396818 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.397099 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.397264 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20521,"thread_start_us":20370,"threads_started":1}
I20260926 23:19:48.397709 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.368075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.0.0.1:44578)
0926 23:19:48.388842 (+ 20767us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.388864 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:48.388966 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:48.389883 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:48.389890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:48.390231 (+ 341us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:48.391019 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.391036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.393029 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.393034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.395276 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.395288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.396640 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.396647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.396875 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.396919 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 23:19:48.397062 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 23:19:48.397267 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20636,"mutex_wait_us":76,"thread_start_us":19309,"threads_started":1}
I20260926 23:19:48.397809 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.367354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:37391 (local address 127.0.0.1:44938)
0926 23:19:48.383411 (+ 16057us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.383481 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:48.383697 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:48.389924 (+ 6227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:48.389929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:48.390278 (+ 349us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:48.390918 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.390933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.392709 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.392716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.395014 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.395028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.396938 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.396944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.397094 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.397149 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 23:19:48.397222 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 23:19:48.397360 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15801,"mutex_wait_us":37,"spinlock_wait_cycles":706176,"thread_start_us":15649,"threads_started":1}
I20260926 23:19:48.398774 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.367617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.16.62.66:37391)
0926 23:19:48.389424 (+ 21807us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.389432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.389458 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:19:48.389550 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.389557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.389744 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.389936 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:48.391197 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.392535 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.395204 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.396690 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.398022 (+ 1332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.398216 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.398285 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.398450 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21644,"thread_start_us":21652,"threads_started":1}
I20260926 23:19:48.400427 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.367632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:32949 (local address 127.0.0.1:53202)
0926 23:19:48.387732 (+ 20100us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.387755 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:48.387896 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:48.392569 (+ 4673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:48.392577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:48.392607 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:48.393158 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.393175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.395126 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.395130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.397251 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.397263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.399590 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.399596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.399768 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.399805 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:19:48.399904 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 23:19:48.400026 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19915,"mutex_wait_us":26,"thread_start_us":20949,"threads_started":1}
I20260926 23:19:48.400436 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.368415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.16.62.67:32949)
0926 23:19:48.391990 (+ 23575us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.392000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.392030 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:19:48.392117 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.392124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.392377 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.392549 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:48.393357 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.394979 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.397410 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.399124 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.399266 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.399500 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.399968 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.400110 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23390,"thread_start_us":23751,"threads_started":1}
I20260926 23:19:48.402288 31159 tablet_service.cc:1467] Processing CreateTablet for tablet 4577d0397ecf49b98bcd69b7651c0abf (DEFAULT_TABLE table=test-table [id=51f2ca09ef184b4c98ed53703872f868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:19:48.403246 31291 tablet_service.cc:1467] Processing CreateTablet for tablet 4577d0397ecf49b98bcd69b7651c0abf (DEFAULT_TABLE table=test-table [id=51f2ca09ef184b4c98ed53703872f868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:19:48.403249 31422 tablet_service.cc:1467] Processing CreateTablet for tablet 4577d0397ecf49b98bcd69b7651c0abf (DEFAULT_TABLE table=test-table [id=51f2ca09ef184b4c98ed53703872f868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:19:48.404474 31159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4577d0397ecf49b98bcd69b7651c0abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:48.404613 31291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4577d0397ecf49b98bcd69b7651c0abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:48.404627 31422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4577d0397ecf49b98bcd69b7651c0abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:19:48.416424 31503 tablet_bootstrap.cc:492] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: Bootstrap starting.
I20260926 23:19:48.417008 31224 heartbeater.cc:503] Master 127.16.62.126:37363 was elected leader, sending a full tablet report...
I20260926 23:19:48.420077 31503 tablet_bootstrap.cc:654] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:48.420905 31504 tablet_bootstrap.cc:492] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: Bootstrap starting.
I20260926 23:19:48.421087 31505 tablet_bootstrap.cc:492] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Bootstrap starting.
I20260926 23:19:48.421456 31503 log.cc:824] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:48.424463 31504 tablet_bootstrap.cc:654] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:48.424510 31505 tablet_bootstrap.cc:654] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Neither blocks nor log segments found. Creating new log.
I20260926 23:19:48.424714 31503 tablet_bootstrap.cc:492] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: No bootstrap required, opened a new log
I20260926 23:19:48.424985 31503 ts_tablet_manager.cc:1397] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 23:19:48.425769 31504 log.cc:824] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:48.426363 31505 log.cc:824] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:48.428369 31504 tablet_bootstrap.cc:492] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: No bootstrap required, opened a new log
I20260926 23:19:48.428658 31504 ts_tablet_manager.cc:1397] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 23:19:48.430655 31505 tablet_bootstrap.cc:492] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: No bootstrap required, opened a new log
I20260926 23:19:48.430938 31505 ts_tablet_manager.cc:1397] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260926 23:19:48.436280 31503 raft_consensus.cc:348] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } }
I20260926 23:19:48.436941 31503 raft_consensus.cc:374] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:48.437067 31503 raft_consensus.cc:729] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79be2c930ef409e9f9fff03c0c521ee, State: Initialized, Role: FOLLOWER
I20260926 23:19:48.436951 31504 raft_consensus.cc:348] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } }
I20260926 23:19:48.437422 31504 raft_consensus.cc:374] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:48.437561 31504 raft_consensus.cc:729] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e39727907a4bb9becc5838a2172e3e, State: Initialized, Role: FOLLOWER
I20260926 23:19:48.437808 31503 consensus_queue.cc:260] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } }
I20260926 23:19:48.438187 31504 consensus_queue.cc:260] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } }
I20260926 23:19:48.439205 31356 heartbeater.cc:503] Master 127.16.62.126:37363 was elected leader, sending a full tablet report...
I20260926 23:19:48.439776 31503 ts_tablet_manager.cc:1428] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 23:19:48.440712 31487 heartbeater.cc:503] Master 127.16.62.126:37363 was elected leader, sending a full tablet report...
I20260926 23:19:48.441989 31505 raft_consensus.cc:348] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } }
I20260926 23:19:48.442436 31505 raft_consensus.cc:374] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:19:48.442551 31505 raft_consensus.cc:729] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af571e43ea33442b911da61ee60015ee, State: Initialized, Role: FOLLOWER
I20260926 23:19:48.443176 31505 consensus_queue.cc:260] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } }
I20260926 23:19:48.443506 31504 ts_tablet_manager.cc:1428] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260926 23:19:48.445006 31505 ts_tablet_manager.cc:1428] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260926 23:19:48.458244 31509 raft_consensus.cc:482] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:19:48.458561 31509 raft_consensus.cc:504] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } }
I20260926 23:19:48.460542 31509 leader_election.cc:288] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e39727907a4bb9becc5838a2172e3e (127.16.62.67:32949), af571e43ea33442b911da61ee60015ee (127.16.62.65:37859)
I20260926 23:19:48.467782 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.461034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:52965 (local address 127.16.62.67:32949)
0926 23:19:48.461285 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.461294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.462002 (+ 708us) server_negotiation.cc:407] Connection header received
0926 23:19:48.462208 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.462213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.462271 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.462371 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:48.462943 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.463679 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.466008 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.466785 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.466976 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.467059 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.467375 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.467511 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:19:48.467783 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.461116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:46487 (local address 127.16.62.65:37859)
0926 23:19:48.461341 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.461350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.462088 (+ 738us) server_negotiation.cc:407] Connection header received
0926 23:19:48.462281 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.462285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.462343 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.462437 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:19:48.463105 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.463821 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.466021 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.466791 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.466980 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.467067 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.467378 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.467512 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:19:48.467945 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.460965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:32949 (local address 127.16.62.66:52965)
0926 23:19:48.461891 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.461908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:48.462076 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:48.462385 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:48.462389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:48.462412 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:48.462789 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.462798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.463807 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.463820 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.465853 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.465867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.467030 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.467037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.467219 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.467243 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 23:19:48.467349 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 23:19:48.467526 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":118,"thread_start_us":613,"threads_started":1}
I20260926 23:19:48.468242 31512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.461055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.16.62.66:46487)
0926 23:19:48.462003 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.462018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:19:48.462131 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:19:48.462474 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:19:48.462478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:19:48.462517 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:19:48.462942 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.462950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.463969 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.463976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.465876 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:19:48.465888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.467066 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.467073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:19:48.467225 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.467248 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:19:48.467351 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 23:19:48.467512 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":26,"thread_start_us":437,"threads_started":1}
I20260926 23:19:48.469062 31442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4577d0397ecf49b98bcd69b7651c0abf" candidate_uuid: "f79be2c930ef409e9f9fff03c0c521ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e39727907a4bb9becc5838a2172e3e" is_pre_election: true
I20260926 23:19:48.469164 31179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4577d0397ecf49b98bcd69b7651c0abf" candidate_uuid: "f79be2c930ef409e9f9fff03c0c521ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af571e43ea33442b911da61ee60015ee" is_pre_election: true
I20260926 23:19:48.469569 31442 raft_consensus.cc:2454] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f79be2c930ef409e9f9fff03c0c521ee in term 0.
I20260926 23:19:48.469807 31179 raft_consensus.cc:2454] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f79be2c930ef409e9f9fff03c0c521ee in term 0.
I20260926 23:19:48.470234 31245 leader_election.cc:302] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e39727907a4bb9becc5838a2172e3e, f79be2c930ef409e9f9fff03c0c521ee; no voters:
I20260926 23:19:48.470623 31509 raft_consensus.cc:2781] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:19:48.470769 31509 raft_consensus.cc:482] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:19:48.470883 31509 raft_consensus.cc:3037] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:48.473248 31509 raft_consensus.cc:504] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } }
I20260926 23:19:48.473905 31509 leader_election.cc:288] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [CANDIDATE]: Term 1 election: Requested vote from peers 13e39727907a4bb9becc5838a2172e3e (127.16.62.67:32949), af571e43ea33442b911da61ee60015ee (127.16.62.65:37859)
I20260926 23:19:48.474299 31442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4577d0397ecf49b98bcd69b7651c0abf" candidate_uuid: "f79be2c930ef409e9f9fff03c0c521ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e39727907a4bb9becc5838a2172e3e"
I20260926 23:19:48.474413 31179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4577d0397ecf49b98bcd69b7651c0abf" candidate_uuid: "f79be2c930ef409e9f9fff03c0c521ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af571e43ea33442b911da61ee60015ee"
I20260926 23:19:48.474498 31442 raft_consensus.cc:3037] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:48.474653 31179 raft_consensus.cc:3037] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:19:48.477005 31442 raft_consensus.cc:2454] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f79be2c930ef409e9f9fff03c0c521ee in term 1.
I20260926 23:19:48.477012 31179 raft_consensus.cc:2454] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f79be2c930ef409e9f9fff03c0c521ee in term 1.
I20260926 23:19:48.477416 31244 leader_election.cc:302] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af571e43ea33442b911da61ee60015ee, f79be2c930ef409e9f9fff03c0c521ee; no voters:
I20260926 23:19:48.477700 31509 raft_consensus.cc:2781] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:19:48.478149 31509 raft_consensus.cc:686] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 1 LEADER]: Becoming Leader. State: Replica: f79be2c930ef409e9f9fff03c0c521ee, State: Running, Role: LEADER
I20260926 23:19:48.478600 31509 consensus_queue.cc:237] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } }
I20260926 23:19:48.483851 31038 catalog_manager.cc:5495] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee reported cstate change: term changed from 0 to 1, leader changed from <none> to f79be2c930ef409e9f9fff03c0c521ee (127.16.62.66). New cstate: current_term: 1 leader_uuid: "f79be2c930ef409e9f9fff03c0c521ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f79be2c930ef409e9f9fff03c0c521ee" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 37391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 } health_report { overall_health: UNKNOWN } } }
I20260926 23:19:48.521271 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.515926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.16.62.66:37391)
0926 23:19:48.516112 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.516117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.516186 (+ 69us) server_negotiation.cc:407] Connection header received
0926 23:19:48.516367 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.516371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.516447 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.516572 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:19:48.517017 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.517775 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.519216 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.519598 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.519693 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.520226 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:19:48.520732 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:19:48.520844 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.520943 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.521042 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:19:48.529094 31179 raft_consensus.cc:1261] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [term 1 FOLLOWER]: Refusing update from remote peer f79be2c930ef409e9f9fff03c0c521ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:19:48.529309 31442 raft_consensus.cc:1261] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [term 1 FOLLOWER]: Refusing update from remote peer f79be2c930ef409e9f9fff03c0c521ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:19:48.530061 31514 consensus_queue.cc:1035] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "af571e43ea33442b911da61ee60015ee" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:19:48.530421 31509 consensus_queue.cc:1035] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e39727907a4bb9becc5838a2172e3e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 32949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:19:48.556787 31038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:19:48.557091 31038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:19:48.560917 31038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Sending DeleteTablet for 3 replicas of tablet 4577d0397ecf49b98bcd69b7651c0abf
I20260926 23:19:48.562171 31291 tablet_service.cc:1514] Processing DeleteTablet for tablet 4577d0397ecf49b98bcd69b7651c0abf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:19:48 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:44938
I20260926 23:19:48.562804 31422 tablet_service.cc:1514] Processing DeleteTablet for tablet 4577d0397ecf49b98bcd69b7651c0abf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:19:48 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:53202
I20260926 23:19:48.563138 31527 tablet_replica.cc:330] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: stopping tablet replica
I20260926 23:19:48.563570 31527 raft_consensus.cc:2229] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:19:48.563715 31528 tablet_replica.cc:330] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: stopping tablet replica
I20260926 23:19:48.564079 31159 tablet_service.cc:1514] Processing DeleteTablet for tablet 4577d0397ecf49b98bcd69b7651c0abf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:19:48 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:44578
I20260926 23:19:48.564237 31528 raft_consensus.cc:2229] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:19:48.564419 31527 raft_consensus.cc:2258] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:19:48.564725 31528 raft_consensus.cc:2258] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:19:48.565797 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 31076
I20260926 23:19:48.567389 31528 ts_tablet_manager.cc:1905] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:19:48.568671 31527 ts_tablet_manager.cc:1905] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:19:48.575873 31529 tablet_replica.cc:330] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: stopping tablet replica
I20260926 23:19:48.576413 31529 raft_consensus.cc:2229] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:19:48.576972 31529 raft_consensus.cc:2258] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:19:48.579140 31529 ts_tablet_manager.cc:1905] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:19:48.586122 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.567039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.16.62.65:37859)
0926 23:19:48.567250 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.567258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.567288 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:19:48.567358 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.567365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.567478 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.567608 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:48.569557 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.571140 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.573168 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.574106 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.574263 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.578747 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:48.578760 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:48.578777 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:48.578814 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:48.582661 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.583590 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.583598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.583602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.583764 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.584232 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.584238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.584243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.584753 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:48.585074 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.585375 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.585604 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:19:48.586637 31528 ts_tablet_manager.cc:1918] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:48.586870 31219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:48.586884 31528 log.cc:1196] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/4577d0397ecf49b98bcd69b7651c0abf
I20260926 23:19:48.587038 31527 ts_tablet_manager.cc:1918] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:48.587239 31527 log.cc:1196] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/4577d0397ecf49b98bcd69b7651c0abf
I20260926 23:19:48.587488 31528 ts_tablet_manager.cc:1939] T 4577d0397ecf49b98bcd69b7651c0abf P 13e39727907a4bb9becc5838a2172e3e: Deleting consensus metadata
I20260926 23:19:48.588027 31527 ts_tablet_manager.cc:1939] T 4577d0397ecf49b98bcd69b7651c0abf P f79be2c930ef409e9f9fff03c0c521ee: Deleting consensus metadata
I20260926 23:19:48.589608 31024 catalog_manager.cc:4841] TS 13e39727907a4bb9becc5838a2172e3e (127.16.62.67:32949): tablet 4577d0397ecf49b98bcd69b7651c0abf (table test-table [id=51f2ca09ef184b4c98ed53703872f868]) successfully deleted
I20260926 23:19:48.590163 31026 catalog_manager.cc:4841] TS f79be2c930ef409e9f9fff03c0c521ee (127.16.62.66:37391): tablet 4577d0397ecf49b98bcd69b7651c0abf (table test-table [id=51f2ca09ef184b4c98ed53703872f868]) successfully deleted
I20260926 23:19:48.586575 31529 ts_tablet_manager.cc:1918] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:48.592427 31529 log.cc:1196] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/4577d0397ecf49b98bcd69b7651c0abf
I20260926 23:19:48.710901 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 31076
I20260926 23:19:48.710906 31529 ts_tablet_manager.cc:1939] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Deleting consensus metadata
W20260926 23:19:48.728826 31023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.62.65:37859 (error 108)
I20260926 23:19:48.729099 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 31228
I20260926 23:19:48.731215 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.730740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.0.0.1:44600)
0926 23:19:48.730916 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.731036 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:19:48.731469 31023 catalog_manager.cc:4584] TS af571e43ea33442b911da61ee60015ee (127.16.62.65:37859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4577d0397ecf49b98bcd69b7651c0abf: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37859: connect: Connection refused (error 111)
I20260926 23:19:48.751127 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.729915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.16.62.66:37391)
0926 23:19:48.730117 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.730124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.730255 (+ 131us) server_negotiation.cc:407] Connection header received
0926 23:19:48.730396 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.730402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.730490 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.730614 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:48.732265 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.733521 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.735085 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.735817 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.735945 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.743556 (+ 7611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:48.743578 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:48.743599 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:48.743664 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:48.748166 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.748941 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.748951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.748955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.749138 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.749516 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.749521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.749525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.749990 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:48.750344 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.750498 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.750721 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:19:48.751940 31351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:48.785382 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.784802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.0.0.1:44610)
0926 23:19:48.785060 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.785198 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:19:48.786577 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.786180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.0.0.1:44616)
0926 23:19:48.786357 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.786451 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 23:19:48.847048 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.846586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.0.0.1:44618)
0926 23:19:48.846787 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.846876 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 23:19:48.848197 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.847786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.0.0.1:44626)
0926 23:19:48.847951 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.848026 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 23:19:48.867304 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 31228
I20260926 23:19:48.886646 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 31359
I20260926 23:19:48.899847 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.887440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.16.62.67:32949)
0926 23:19:48.887611 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:19:48.887615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:19:48.887633 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:19:48.887797 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:19:48.887801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:19:48.887865 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:19:48.887951 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:19:48.889360 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.890079 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.891377 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:19:48.891771 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:19:48.891866 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:19:48.894405 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:19:48.894420 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:19:48.894435 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:19:48.894475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:19:48.897332 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.898011 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.898018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.898020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.898205 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:19:48.898579 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:19:48.898583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:19:48.898585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:19:48.898945 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:19:48.899155 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 23:19:48.899421 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 23:19:48.899585 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:19:48.900532 31482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:48.928881 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.928405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.0.0.1:44632)
0926 23:19:48.928608 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.928716 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 23:19:48.930001 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:48.929603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37859 (local address 127.0.0.1:44642)
0926 23:19:48.929824 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:48.929894 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 23:19:49.000115 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 31359
I20260926 23:19:49.017895 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 31006
I20260926 23:19:49.019186 31069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:19:49.132423 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 31006
I20260926 23:19:49.156028 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:37363
--webserver_interface=127.16.62.126
--webserver_port=33819
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:37363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:19:49.489415 31535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:49.489692 31535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:49.489861 31535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:49.496974 31535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:19:49.497074 31535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:49.497169 31535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:49.497284 31535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:19:49.497364 31535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:19:49.506947 31535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:37363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:37363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=33819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:49.508322 31535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:49.509640 31535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:49.521864 31540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:49.521826 31541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:49.521898 31543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:49.522675 31535 server_base.cc:1034] running on GCE node
I20260926 23:19:49.523427 31535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:49.524951 31535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:49.526209 31535 hybrid_clock.cc:648] HybridClock initialized: now 1790464789526118 us; error 56 us; skew 500 ppm
I20260926 23:19:49.526595 31535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:49 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464789, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:49.529919 31535 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:19:49.531620 31535 webserver.cc:460] Webserver started at http://127.16.62.126:33819/ using document root <none> and password file <none>
I20260926 23:19:49.532456 31535 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:49.532580 31535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:49.538439 31535 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 23:19:49.541517 31550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:49.542949 31535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:19:49.543181 31535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "1577316473b446d2a7a76c1b7263b81c"
format_stamp: "Formatted at 2026-09-26 23:19:43 on dist-test-slave-vcv1"
server_key: "1312e3e16f4522f5ac8685e4d90f425a"
server_key_iv: "6ca3b15264759a07c141893cb8376ccf"
server_key_version: "encryptionkey@0"
I20260926 23:19:49.544251 31535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:49.558772 31535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:49.563184 31535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:49.563488 31535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:49.582466 31535 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:37363
I20260926 23:19:49.582504 31601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:37363 every 8 connection(s)
I20260926 23:19:49.584007 31535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:19:49.591118 31602 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:19:49.593712 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 31535
I20260926 23:19:49.594771 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:37859
--local_ip_for_outbound_sockets=127.16.62.65
--tserver_master_addrs=127.16.62.126:37363
--webserver_port=44729
--webserver_interface=127.16.62.65
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:19:49.595827 31602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Bootstrap starting.
I20260926 23:19:49.619540 31602 log.cc:824] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:19:49.637233 31602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:19:49.637900 31602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c: Bootstrap complete.
I20260926 23:19:49.648208 31602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:49.649384 31602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1577316473b446d2a7a76c1b7263b81c, State: Initialized, Role: FOLLOWER
I20260926 23:19:49.650074 31602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:49.650326 31602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:19:49.650481 31602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:19:49.650682 31602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 2 FOLLOWER]: Advancing to term 3
I20260926 23:19:49.654837 31602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:49.655357 31602 leader_election.cc:302] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1577316473b446d2a7a76c1b7263b81c; no voters:
I20260926 23:19:49.656106 31602 leader_election.cc:288] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 23:19:49.656279 31606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 3 FOLLOWER]: Leader election won for term 3
I20260926 23:19:49.657590 31606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [term 3 LEADER]: Becoming Leader. State: Replica: 1577316473b446d2a7a76c1b7263b81c, State: Running, Role: LEADER
I20260926 23:19:49.658283 31606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } }
I20260926 23:19:49.658576 31602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:19:49.661156 31607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1577316473b446d2a7a76c1b7263b81c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } } }
I20260926 23:19:49.661634 31607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:49.663962 31608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1577316473b446d2a7a76c1b7263b81c. Latest consensus state: current_term: 3 leader_uuid: "1577316473b446d2a7a76c1b7263b81c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1577316473b446d2a7a76c1b7263b81c" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 37363 } } }
I20260926 23:19:49.664361 31608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1577316473b446d2a7a76c1b7263b81c [sys.catalog]: This master's current role is: LEADER
I20260926 23:19:49.666272 31612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:19:49.676349 31612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:19:49.677948 31612 catalog_manager.cc:1259] Loaded cluster ID: ad04a3227393495ca10c46ec272dd4fd
I20260926 23:19:49.678114 31612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 23:19:49.680302 31612 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3ccf2a69ad google::LogMessage::Fail() at ??:0
@ 0x7f3ccf2a988a google::LogMessage::SendToLog() at ??:0
@ 0x7f3ccf2a64de google::LogMessage::Flush() at ??:0
@ 0x7f3ccf2a7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3ce4cd9512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3ce4cd6976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3ce4d5e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3ce4d5e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3cd1ef2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3cd1ef591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3cd1ef5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3cd1ed0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3cd51aa6db start_thread at ??:0
@ 0x7f3cce14061f clone at ??:0
W20260926 23:19:49.999600 31604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:49.999871 31604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:49.999965 31604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:50.007165 31604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:50.007259 31604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:50.007364 31604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:19:50.016878 31604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:37859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=44729
--tserver_master_addrs=127.16.62.126:37363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:50.018079 31604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:50.019337 31604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:50.032383 31644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:50.032579 31643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:50.032457 31646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:50.033530 31604 server_base.cc:1034] running on GCE node
I20260926 23:19:50.034154 31604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:50.035086 31604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:50.036319 31604 hybrid_clock.cc:648] HybridClock initialized: now 1790464790036268 us; error 45 us; skew 500 ppm
I20260926 23:19:50.036648 31604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:50 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:50.039696 31604 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:19:50.041172 31604 webserver.cc:460] Webserver started at http://127.16.62.65:44729/ using document root <none> and password file <none>
I20260926 23:19:50.041857 31604 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:50.042002 31604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:50.047425 31604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:19:50.050239 31653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:50.051608 31604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:19:50.051800 31604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "af571e43ea33442b911da61ee60015ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "6a165683996ae1c93a84708de8d2a67e"
server_key_iv: "f67108af21ed97032685c19bd55a299f"
server_key_version: "encryptionkey@0"
I20260926 23:19:50.052920 31604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:50.069579 31604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:50.073752 31604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:50.074030 31604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:50.075017 31604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:50.076864 31660 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 23:19:50.082044 31660 ts_tablet_manager.cc:1852] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 23:19:50.083005 31660 ts_tablet_manager.cc:1905] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:19:50.090116 31660 ts_tablet_manager.cc:1918] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:19:50.090346 31660 ts_tablet_manager.cc:1939] T 4577d0397ecf49b98bcd69b7651c0abf P af571e43ea33442b911da61ee60015ee: Deleting consensus metadata
I20260926 23:19:50.090794 31604 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 23:19:50.090988 31604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260926 23:19:50.091158 31604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:50.091262 31604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:50.127202 31604 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:37859
I20260926 23:19:50.127259 31766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:37859 every 8 connection(s)
I20260926 23:19:50.128664 31604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:19:50.133823 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:50.131888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:52111)
0926 23:19:50.132675 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:50.132919 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":334,"threads_started":1}
W20260926 23:19:50.134423 31767 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:37363 (0 consecutive failures): Network error: Failed to ping master at 127.16.62.126:37363: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
I20260926 23:19:50.134447 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 31604
I20260926 23:19:50.135218 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:37391
--local_ip_for_outbound_sockets=127.16.62.66
--tserver_master_addrs=127.16.62.126:37363
--webserver_port=33119
--webserver_interface=127.16.62.66
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:19:50.135371 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:50.134956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:40451)
0926 23:19:50.135143 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:50.135235 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 23:19:50.136363 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:50.135971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:57535)
0926 23:19:50.136109 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:50.136249 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 23:19:50.136667 31767 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:19:50.461072 31770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:50.461359 31770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:50.461519 31770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:50.468717 31770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:50.468815 31770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:50.469017 31770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:19:50.478645 31770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:37391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=33119
--tserver_master_addrs=127.16.62.126:37363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:50.479950 31770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:50.481262 31770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:50.494233 31775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:50.494278 31778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:50.494329 31776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:50.497479 31770 server_base.cc:1034] running on GCE node
I20260926 23:19:50.498018 31770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:50.498922 31770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:50.500118 31770 hybrid_clock.cc:648] HybridClock initialized: now 1790464790500076 us; error 31 us; skew 500 ppm
I20260926 23:19:50.500434 31770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:50 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:50.503391 31770 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:19:50.504902 31770 webserver.cc:460] Webserver started at http://127.16.62.66:33119/ using document root <none> and password file <none>
I20260926 23:19:50.505615 31770 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:50.505715 31770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:50.510818 31770 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:19:50.513550 31785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:50.514881 31770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:19:50.515074 31770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "f79be2c930ef409e9f9fff03c0c521ee"
format_stamp: "Formatted at 2026-09-26 23:19:44 on dist-test-slave-vcv1"
server_key: "ff545c82ada71149ccc594aea3e19eac"
server_key_iv: "47674d2cfee15b95ae851238dad2473e"
server_key_version: "encryptionkey@0"
I20260926 23:19:50.516086 31770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:50.531056 31770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:50.535439 31770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:50.535820 31770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:50.536985 31770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:50.538534 31770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:50.538641 31770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:50.538807 31770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:50.538909 31770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:50.575488 31770 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:37391
I20260926 23:19:50.575584 31897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:37391 every 8 connection(s)
I20260926 23:19:50.577104 31770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:19:50.578732 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 31770
I20260926 23:19:50.579556 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:32949
--local_ip_for_outbound_sockets=127.16.62.67
--tserver_master_addrs=127.16.62.126:37363
--webserver_port=39343
--webserver_interface=127.16.62.67
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:19:50.582499 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:50.580623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:47185)
0926 23:19:50.581427 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:50.581632 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":344,"threads_started":1}
W20260926 23:19:50.583326 31898 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:37363 (0 consecutive failures): Network error: Failed to ping master at 127.16.62.126:37363: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
I20260926 23:19:50.584283 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:50.583890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:59577)
0926 23:19:50.584044 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:50.584114 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 23:19:50.585237 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:50.584920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:43583)
0926 23:19:50.585050 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:50.585133 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 23:19:50.585534 31898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:19:50.906821 31901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:19:50.907099 31901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:19:50.907191 31901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:19:50.914471 31901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:19:50.914592 31901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:19:50.914713 31901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:19:50.924999 31901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:32949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=39343
--tserver_master_addrs=127.16.62.126:37363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:19:50.926261 31901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:19:50.927518 31901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:19:50.940728 31909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:50.940783 31906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:19:50.940752 31907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:19:50.941478 31901 server_base.cc:1034] running on GCE node
I20260926 23:19:50.942168 31901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:19:50.943113 31901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:19:50.944299 31901 hybrid_clock.cc:648] HybridClock initialized: now 1790464790944262 us; error 34 us; skew 500 ppm
I20260926 23:19:50.944612 31901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:19:50 dist-test-slave-vcv1 krb5kdc[30457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:19:50.947980 31901 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:19:50.949420 31901 webserver.cc:460] Webserver started at http://127.16.62.67:39343/ using document root <none> and password file <none>
I20260926 23:19:50.950110 31901 fs_manager.cc:359] Metadata directory not provided
I20260926 23:19:50.950258 31901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:19:50.955655 31901 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:19:50.958518 31916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:50.959844 31901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:19:50.960016 31901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "13e39727907a4bb9becc5838a2172e3e"
format_stamp: "Formatted at 2026-09-26 23:19:45 on dist-test-slave-vcv1"
server_key: "50cf1f0f22920d18416611a21c1f9a45"
server_key_iv: "a15e558fe42a351b8885a24f10c97786"
server_key_version: "encryptionkey@0"
I20260926 23:19:50.961053 31901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:19:50.980109 31901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:19:50.984542 31901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:19:50.984836 31901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:19:50.985922 31901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:19:50.987496 31901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:19:50.987591 31901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:50.987761 31901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:19:50.987855 31901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:19:51.021394 31901 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:32949
I20260926 23:19:51.021485 32028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:32949 every 8 connection(s)
I20260926 23:19:51.022855 31901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:19:51.024425 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 31901
I20260926 23:19:51.027644 32032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:51.025892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:56767)
0926 23:19:51.026637 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:51.026849 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":347,"threads_started":1}
W20260926 23:19:51.028390 32029 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:37363 (0 consecutive failures): Network error: Failed to ping master at 127.16.62.126:37363: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
I20260926 23:19:51.029153 32032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:51.028844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:36449)
0926 23:19:51.028991 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:51.029058 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 23:19:51.029959 32032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:51.029644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:33755)
0926 23:19:51.029780 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:51.029857 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:19:51.030182 32029 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:19:51.138052 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:51.137152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:48461)
0926 23:19:51.137750 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:51.137825 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":316,"threads_started":1}
I20260926 23:19:51.586802 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:51.586027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:58259)
0926 23:19:51.586468 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:51.586580 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":209,"threads_started":1}
I20260926 23:19:52.031625 32035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:52.030800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:60589)
0926 23:19:52.031280 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:52.031388 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":239,"threads_started":1}
I20260926 23:19:52.139726 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:52.138968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:49197)
0926 23:19:52.139471 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:52.139560 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":274,"threads_started":1}
I20260926 23:19:52.588524 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:52.587716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:34281)
0926 23:19:52.588278 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:52.588360 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":237,"threads_started":1}
I20260926 23:19:53.033432 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:53.032705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:53999)
0926 23:19:53.033192 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:53.033272 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":244,"threads_started":1}
I20260926 23:19:53.141263 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:53.140593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:48685)
0926 23:19:53.141018 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:53.141106 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":215,"threads_started":1}
I20260926 23:19:53.590219 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:53.589421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:60755)
0926 23:19:53.589921 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:53.590027 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":232,"threads_started":1}
I20260926 23:19:54.035259 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:54.034288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:49793)
0926 23:19:54.035016 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:54.035091 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":553,"threads_started":1}
I20260926 23:19:54.142874 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:54.142111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:45719)
0926 23:19:54.142605 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:54.142693 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":249,"threads_started":1}
I20260926 23:19:54.591781 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:54.591070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:34899)
0926 23:19:54.591485 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:54.591568 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":212,"threads_started":1}
I20260926 23:19:55.036993 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:55.036067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:52721)
0926 23:19:55.036609 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:55.036726 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":297,"threads_started":1}
I20260926 23:19:55.144567 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:55.143788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:54295)
0926 23:19:55.144325 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:55.144409 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":253,"threads_started":1}
I20260926 23:19:55.593678 32046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:55.592853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:48843)
0926 23:19:55.593433 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:55.593509 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":342,"threads_started":1}
I20260926 23:19:56.038620 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:56.037864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:36773)
0926 23:19:56.038348 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:56.038441 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
I20260926 23:19:56.146432 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:56.145665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:44411)
0926 23:19:56.146169 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:56.146256 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
I20260926 23:19:56.595461 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:56.594690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:43119)
0926 23:19:56.595208 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:56.595295 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":273,"threads_started":1}
I20260926 23:19:57.040187 32050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:57.039467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:41817)
0926 23:19:57.039921 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:57.040008 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":249,"threads_started":1}
I20260926 23:19:57.148470 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:57.147471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:43197)
0926 23:19:57.148160 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:57.148261 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":328,"threads_started":1}
I20260926 23:19:57.597424 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:57.596601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:56989)
0926 23:19:57.597123 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:57.597205 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":232,"threads_started":1}
I20260926 23:19:58.042013 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:58.041204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:58483)
0926 23:19:58.041707 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:58.041794 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":235,"threads_started":1}
I20260926 23:19:58.150050 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:58.149334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:34397)
0926 23:19:58.149755 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:58.149837 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":226,"threads_started":1}
I20260926 23:19:58.599040 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:58.598339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:40125)
0926 23:19:58.598794 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:58.598878 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":222,"threads_started":1}
I20260926 23:19:59.043783 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:59.043039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:35491)
0926 23:19:59.043544 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:59.043622 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":264,"threads_started":1}
I20260926 23:19:59.151655 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:59.150930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:44527)
0926 23:19:59.151394 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:59.151477 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":229,"threads_started":1}
I20260926 23:19:59.600768 32058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:19:59.599931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:49543)
0926 23:19:59.600503 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 23:19:59.600601 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":312,"threads_started":1}
I20260926 23:20:00.045590 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:00.044772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:48831)
0926 23:20:00.045344 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:00.045422 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":268,"threads_started":1}
I20260926 23:20:00.153343 32060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:00.152672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:42823)
0926 23:20:00.153077 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:00.153168 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260926 23:20:00.602552 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:00.601779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:43361)
0926 23:20:00.602320 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:00.602397 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":337,"threads_started":1}
I20260926 23:20:01.047360 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:01.046558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:40029)
0926 23:20:01.047030 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:01.047204 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":293,"threads_started":1}
I20260926 23:20:01.154953 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:01.154253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:51241)
0926 23:20:01.154693 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:01.154770 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":249,"threads_started":1}
I20260926 23:20:01.604110 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:01.603407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:56929)
0926 23:20:01.603841 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:01.603928 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260926 23:20:02.049072 32065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:02.048309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:33989)
0926 23:20:02.048792 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:02.048893 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":258,"threads_started":1}
I20260926 23:20:02.156836 32066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:02.155941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:50615)
0926 23:20:02.156465 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:02.156595 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":326,"threads_started":1}
I20260926 23:20:02.605903 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:02.605159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:59085)
0926 23:20:02.605660 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:02.605744 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260926 23:20:03.051144 32068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:03.050185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:40423)
0926 23:20:03.050791 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:03.050911 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":293,"threads_started":1}
I20260926 23:20:03.158420 32069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:03.157687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:59085)
0926 23:20:03.158154 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:03.158253 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260926 23:20:03.607530 32070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:03.606771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:46853)
0926 23:20:03.607244 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:03.607347 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":251,"threads_started":1}
I20260926 23:20:04.053013 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:04.052291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:56433)
0926 23:20:04.052748 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:04.052837 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":220,"threads_started":1}
I20260926 23:20:04.159979 32072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:04.159240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:37657)
0926 23:20:04.159694 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:04.159782 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":223,"threads_started":1}
I20260926 23:20:04.609226 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:04.608512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:42467)
0926 23:20:04.608952 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:04.609033 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
I20260926 23:20:05.054747 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:05.053998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:52649)
0926 23:20:05.054493 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:05.054571 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":275,"threads_started":1}
I20260926 23:20:05.161901 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:05.161068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.65:48717)
0926 23:20:05.161598 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:05.161702 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":255,"threads_started":1}
I20260926 23:20:05.611047 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:05.610289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.66:36991)
0926 23:20:05.610766 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:05.610855 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":255,"threads_started":1}
I20260926 23:20:06.033936 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 31604
I20260926 23:20:06.056890 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:06.055930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:37363 (local address 127.16.62.67:48505)
0926 23:20:06.056551 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:06.056649 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":349,"threads_started":1}
I20260926 23:20:06.059518 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:06.035128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.16.62.65:37859)
0926 23:20:06.035656 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:06.035674 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:06.035900 (+ 226us) server_negotiation.cc:407] Connection header received
0926 23:20:06.037928 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:06.037938 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:06.038355 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:06.038700 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:06.040100 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:06.041330 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:06.042543 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:06.043215 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:06.043450 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:06.052767 (+ 9317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:06.052775 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:06.052790 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:06.052834 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:06.055971 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:06.056805 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:06.056812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:06.056819 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:06.056978 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:06.057529 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:06.057535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:06.057540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:06.058057 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:06.058366 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:06.058820 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 23:20:06.059232 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":230,"threads_started":1}
I20260926 23:20:06.060897 31762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:06.145248 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 31604
I20260926 23:20:06.162319 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 31770
I20260926 23:20:06.180020 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:06.163348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.16.62.66:37391)
0926 23:20:06.163798 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:06.163812 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:06.163852 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:20:06.165577 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:06.165590 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:06.166008 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:06.166347 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:06.167801 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:06.169123 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:06.170436 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:06.171101 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:06.171292 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:06.174002 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:06.174009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:06.174024 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:06.174069 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:06.176921 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:06.177595 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:06.177603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:06.177608 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:06.177764 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:06.178155 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:06.178161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:06.178164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:06.178622 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:06.178935 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:06.179346 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 23:20:06.179735 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":196,"threads_started":1}
I20260926 23:20:06.181566 31893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:06.266208 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 31770
I20260926 23:20:06.283820 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 31901
I20260926 23:20:06.302412 32082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:06.284864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.16.62.67:32949)
0926 23:20:06.285543 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:06.285558 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:06.285599 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:20:06.287247 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:06.287256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:06.287686 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:06.288032 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:06.289472 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:06.290696 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:06.291952 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:06.292679 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:06.292884 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:06.295608 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:06.295616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:06.295631 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:06.295666 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:06.298861 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:06.299615 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:06.299632 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:06.299638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:06.299812 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:06.300261 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:06.300269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:06.300272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:06.300811 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:06.301163 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:06.301662 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 23:20:06.302088 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":294,"threads_started":1}
I20260926 23:20:06.303828 32024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:06.386173 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 31901
2026-09-26T23:20:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24455 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:20:06 dist-test-slave-vcv1 krb5kdc[32086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:20:06 dist-test-slave-vcv1 krb5kdc[32086](info): set up 2 sockets
Sep 26 23:20:06 dist-test-slave-vcv1 krb5kdc[32086](info): commencing operation
krb5kdc: starting...
W20260926 23:20:07.489801 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:20:07 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:20:07.728586 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:39965
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:39965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:20:08.052858 32099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:08.053170 32099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:08.053262 32099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:08.060648 32099 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 23:20:08.060757 32099 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 23:20:08.060796 32099 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 23:20:08.060848 32099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:20:08.060904 32099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:08.060971 32099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:08.061017 32099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:20:08.061064 32099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:20:08.071163 32099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:39965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:39965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:08.072551 32099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:08.073833 32099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:08.086002 32104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:08.086117 32105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:08.086071 32107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:08.089815 32099 server_base.cc:1034] running on GCE node
I20260926 23:20:08.090667 32099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:08.092186 32099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:08.093428 32099 hybrid_clock.cc:648] HybridClock initialized: now 1790464808093371 us; error 34 us; skew 500 ppm
I20260926 23:20:08.093784 32099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:08 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464808, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:08.097038 32099 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:20:08.098568 32099 webserver.cc:460] Webserver started at http://127.16.62.126:37721/ using document root <none> and password file <none>
I20260926 23:20:08.099298 32099 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:08.099436 32099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:08.099799 32099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:20:08.102880 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "03c42eba704743a580e11a213a3a26a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "40de4a8c403b3a97b16fc6504f0ed0cb"
server_key_iv: "8d6117d72cc9d5c2181bda4221c5114e"
server_key_version: "encryptionkey@0"
I20260926 23:20:08.103654 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "03c42eba704743a580e11a213a3a26a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "40de4a8c403b3a97b16fc6504f0ed0cb"
server_key_iv: "8d6117d72cc9d5c2181bda4221c5114e"
server_key_version: "encryptionkey@0"
I20260926 23:20:08.108985 32099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 23:20:08.112530 32114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:08.113889 32099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:20:08.114085 32099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "03c42eba704743a580e11a213a3a26a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "40de4a8c403b3a97b16fc6504f0ed0cb"
server_key_iv: "8d6117d72cc9d5c2181bda4221c5114e"
server_key_version: "encryptionkey@0"
I20260926 23:20:08.114316 32099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:08.132522 32099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:08.136844 32099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:08.137151 32099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:08.156029 32099 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:39965
I20260926 23:20:08.156049 32165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:39965 every 8 connection(s)
I20260926 23:20:08.157471 32099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:20:08.161213 32166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:20:08.163676 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 32099
I20260926 23:20:08.163931 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:20:08.164760 16633 external_mini_cluster.cc:1428] Setting key 6af460a66a1110bd9b45ec7a6524fae1
I20260926 23:20:08.173576 32166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Bootstrap starting.
I20260926 23:20:08.176741 32166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Neither blocks nor log segments found. Creating new log.
I20260926 23:20:08.177733 32166 log.cc:824] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:20:08 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:20:08.180853 32166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: No bootstrap required, opened a new log
I20260926 23:20:08.188364 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:08.167031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.16.62.126:39965)
0926 23:20:08.168286 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:08.168298 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:08.168338 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:20:08.170012 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:08.170042 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:08.170442 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:08.170858 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:08.172395 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:08.173725 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:08.175127 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:08.175809 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:08.176023 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:08.180867 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:08.180878 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:08.180892 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:08.180963 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:08.184132 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:08.184937 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:08.184950 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:08.184956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:08.185133 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:08.185532 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:08.185538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:08.185541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:08.186078 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:08.186400 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:08.186826 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 23:20:08.187275 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":759,"threads_started":1}
I20260926 23:20:08.189007 32166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:08.189471 32166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:20:08.189594 32166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c42eba704743a580e11a213a3a26a0, State: Initialized, Role: FOLLOWER
I20260926 23:20:08.190366 32166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:08.190613 32166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:20:08.190790 32166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:20:08.190976 32166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:20:08.193367 32166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:08.193845 32166 leader_election.cc:302] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03c42eba704743a580e11a213a3a26a0; no voters:
I20260926 23:20:08.194538 32166 leader_election.cc:288] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:20:08.194777 32171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:20:08.195611 32171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 1 LEADER]: Becoming Leader. State: Replica: 03c42eba704743a580e11a213a3a26a0, State: Running, Role: LEADER
I20260926 23:20:08.196245 32171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:08.196895 32166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:20:08.199162 32173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03c42eba704743a580e11a213a3a26a0. Latest consensus state: current_term: 1 leader_uuid: "03c42eba704743a580e11a213a3a26a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } } }
I20260926 23:20:08.199170 32172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03c42eba704743a580e11a213a3a26a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } } }
I20260926 23:20:08.199589 32173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:20:08.199767 32172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:20:08.203771 32177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:20:08.209138 32177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:20:08.218284 32177 catalog_manager.cc:1347] Generated new cluster ID: 450da93f999a45659d7f54aa4f962977
I20260926 23:20:08.218398 32177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:20:08.242108 32177 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:20:08.243466 32177 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:20:08.254352 32177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Generated new TSK 0
I20260926 23:20:08.255527 32177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:20:08.320771 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:39965
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:20:08.656092 32192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:08.656450 32192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:08.656629 32192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:08.663690 32192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:08.663872 32192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:08.664062 32192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:20:08.673836 32192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:39965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:08.675099 32192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:08.676496 32192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:08.689576 32198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:08.689643 32200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:08.689664 32197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:08.690423 32192 server_base.cc:1034] running on GCE node
I20260926 23:20:08.691006 32192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:08.691924 32192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:08.693146 32192 hybrid_clock.cc:648] HybridClock initialized: now 1790464808693094 us; error 49 us; skew 500 ppm
I20260926 23:20:08.693450 32192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:08 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464808, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:08.696754 32192 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:20:08.698288 32192 webserver.cc:460] Webserver started at http://127.16.62.65:36987/ using document root <none> and password file <none>
I20260926 23:20:08.698937 32192 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:08.699035 32192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:08.699267 32192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:20:08.702104 32192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "0e3b17b3f1591a63c914485a24ad28ec"
server_key_iv: "42eed758b86e764845e8e688f51280e7"
server_key_version: "encryptionkey@0"
I20260926 23:20:08.702811 32192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "0e3b17b3f1591a63c914485a24ad28ec"
server_key_iv: "42eed758b86e764845e8e688f51280e7"
server_key_version: "encryptionkey@0"
I20260926 23:20:08.707962 32192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:20:08.711272 32207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:08.712656 32192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:20:08.712817 32192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "0e3b17b3f1591a63c914485a24ad28ec"
server_key_iv: "42eed758b86e764845e8e688f51280e7"
server_key_version: "encryptionkey@0"
I20260926 23:20:08.712982 32192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:08.740114 32192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:08.744201 32192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:08.744496 32192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:08.745538 32192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:20:08.746901 32192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:20:08.746973 32192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:08.747097 32192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:20:08.747160 32192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:08.780952 32192 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:35069
I20260926 23:20:08.781064 32319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:35069 every 8 connection(s)
I20260926 23:20:08.782272 32192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:20:08.787400 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 32192
I20260926 23:20:08.787634 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:20:08.788515 16633 external_mini_cluster.cc:1428] Setting key 24113d99db733049e33e62700e8702c6
Sep 26 23:20:08 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464808, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:20:08.805323 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:08.784867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:33195 (local address 127.16.62.126:39965)
0926 23:20:08.785342 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:08.785348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:08.786323 (+ 975us) server_negotiation.cc:407] Connection header received
0926 23:20:08.788811 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:08.788816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:08.788896 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:08.789028 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:08.790868 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:08.791590 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:08.794191 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:08.794604 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:08.794706 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:08.799160 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:08.799166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:08.799170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:08.799201 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:08.801949 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:08.802753 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:08.802758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:08.802766 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:08.802879 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:08.803324 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:08.803329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:08.803332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:08.803622 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:08.803797 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:08.804835 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0926 23:20:08.805053 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
I20260926 23:20:08.806753 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:08.785452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:33195)
0926 23:20:08.786157 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:08.786196 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:08.788474 (+ 2278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:08.789310 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:08.789318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:08.789954 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:20:08.790666 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:08.790684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:08.791753 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:08.791761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:08.794020 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:08.794053 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:08.794846 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:08.794856 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:08.795209 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:08.796014 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:20:08.796058 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:20:08.798921 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:20:08.802135 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:08.802144 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:08.802160 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:08.802620 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:08.802990 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:08.802996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:08.803000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:08.803182 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:08.803814 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:20:08.803824 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:20:08.804335 (+ 511us) client_negotiation.cc:769] Sending connection context
0926 23:20:08.804679 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 23:20:08.805236 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":267,"threads_started":1}
I20260926 23:20:08.808490 32320 heartbeater.cc:347] Connected to a master server at 127.16.62.126:39965
I20260926 23:20:08.808828 32320 heartbeater.cc:464] Registering TS with master...
I20260926 23:20:08.809723 32320 heartbeater.cc:511] Master 127.16.62.126:39965 requested a full tablet report, sending...
I20260926 23:20:08.812369 32131 ts_manager.cc:194] Registered new tserver with Master: e0f8a7f5c5614bd5b3bc925cc52411a0 (127.16.62.65:35069)
I20260926 23:20:08.814602 32131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:33195
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:20:08.847401 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:39965
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:20:09.167371 32326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:09.167625 32326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:09.167807 32326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:09.175098 32326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:09.175199 32326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:09.175387 32326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:20:09.184881 32326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:39965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:09.186059 32326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:09.187752 32326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:09.200958 32331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:09.200986 32334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:09.200912 32332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:09.201890 32326 server_base.cc:1034] running on GCE node
I20260926 23:20:09.202541 32326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:09.203420 32326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:09.204649 32326 hybrid_clock.cc:648] HybridClock initialized: now 1790464809204604 us; error 34 us; skew 500 ppm
I20260926 23:20:09.204957 32326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:09 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464809, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:09.207988 32326 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:20:09.209414 32326 webserver.cc:460] Webserver started at http://127.16.62.66:42315/ using document root <none> and password file <none>
I20260926 23:20:09.210074 32326 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:09.210222 32326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:09.210601 32326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:20:09.213438 32326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "5bcc77497d6b49ef98c2242add6b8170"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "e5500f1db782a9cf8c5652145f415117"
server_key_iv: "33ca35f637fbc692eddf0bf3130d58de"
server_key_version: "encryptionkey@0"
I20260926 23:20:09.214239 32326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "5bcc77497d6b49ef98c2242add6b8170"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "e5500f1db782a9cf8c5652145f415117"
server_key_iv: "33ca35f637fbc692eddf0bf3130d58de"
server_key_version: "encryptionkey@0"
I20260926 23:20:09.219568 32326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:20:09.222810 32341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:09.224042 32326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:20:09.224228 32326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "5bcc77497d6b49ef98c2242add6b8170"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "e5500f1db782a9cf8c5652145f415117"
server_key_iv: "33ca35f637fbc692eddf0bf3130d58de"
server_key_version: "encryptionkey@0"
I20260926 23:20:09.224385 32326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:09.247218 32326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:09.251494 32326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:09.251807 32326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:09.252866 32326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:20:09.254210 32326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:20:09.254282 32326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:09.254448 32326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:20:09.254521 32326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:09.288332 32326 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:34229
I20260926 23:20:09.288384 32453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:34229 every 8 connection(s)
I20260926 23:20:09.289639 32326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:20:09.299423 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 32326
I20260926 23:20:09.299623 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:20:09.300268 16633 external_mini_cluster.cc:1428] Setting key cf7a25379da883e5a67c783e756b7b3d
Sep 26 23:20:09 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464809, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:20:09.311056 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.292282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:49711 (local address 127.16.62.126:39965)
0926 23:20:09.292525 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:09.292530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:09.293652 (+ 1122us) server_negotiation.cc:407] Connection header received
0926 23:20:09.295496 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:09.295501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:09.295601 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:09.295739 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:09.297319 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.298105 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.300154 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.300901 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.301070 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.305195 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:09.305200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:09.305204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:09.305236 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:09.307961 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:09.308746 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:09.308756 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:09.308759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:09.308882 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:09.309320 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:09.309330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:09.309332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:09.309604 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:09.309762 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:09.310453 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 23:20:09.310694 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 23:20:09.312027 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.292738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:49711)
0926 23:20:09.293458 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:09.293505 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:09.295215 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:09.295968 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:09.295976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:09.296584 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:20:09.297138 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.297154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.298274 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.298281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.299979 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.299993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.301139 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.301144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.301431 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.302245 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:20:09.302276 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:20:09.304978 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:20:09.308122 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:09.308131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:09.308170 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:09.308610 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:09.308998 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:09.309003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:09.309006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:09.309194 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:09.309743 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:20:09.309751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:20:09.310088 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 23:20:09.310353 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 23:20:09.310890 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":302,"threads_started":1}
I20260926 23:20:09.313500 32454 heartbeater.cc:347] Connected to a master server at 127.16.62.126:39965
I20260926 23:20:09.313809 32454 heartbeater.cc:464] Registering TS with master...
I20260926 23:20:09.314499 32454 heartbeater.cc:511] Master 127.16.62.126:39965 requested a full tablet report, sending...
I20260926 23:20:09.316069 32131 ts_manager.cc:194] Registered new tserver with Master: 5bcc77497d6b49ef98c2242add6b8170 (127.16.62.66:34229)
I20260926 23:20:09.317466 32131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:49711
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:20:09.356482 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:39965
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:20:09.701586 32459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:09.701834 32459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:09.701925 32459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:09.709115 32459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:09.709210 32459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:09.709316 32459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:20:09.718713 32459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:39965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:09.719929 32459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:09.721169 32459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:09.734297 32464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:09.734318 32465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:09.734395 32467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:09.735060 32459 server_base.cc:1034] running on GCE node
I20260926 23:20:09.735675 32459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:09.736690 32459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:09.737933 32459 hybrid_clock.cc:648] HybridClock initialized: now 1790464809737886 us; error 62 us; skew 500 ppm
I20260926 23:20:09.738268 32459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:09 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464809, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:09.742579 32459 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:20:09.744478 32459 webserver.cc:460] Webserver started at http://127.16.62.67:33709/ using document root <none> and password file <none>
I20260926 23:20:09.745338 32459 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:09.745474 32459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:09.745819 32459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:20:09.749482 32459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "787e53d925364e7581f8068528df71bd"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "132d6bf632faa60216fb951ebb278014"
server_key_iv: "0dff8567f3e613f8eca4c215bc91138a"
server_key_version: "encryptionkey@0"
I20260926 23:20:09.750247 32459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "787e53d925364e7581f8068528df71bd"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "132d6bf632faa60216fb951ebb278014"
server_key_iv: "0dff8567f3e613f8eca4c215bc91138a"
server_key_version: "encryptionkey@0"
I20260926 23:20:09.755364 32459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 23:20:09.758948 32474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:09.760528 32459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 23:20:09.760699 32459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "787e53d925364e7581f8068528df71bd"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "132d6bf632faa60216fb951ebb278014"
server_key_iv: "0dff8567f3e613f8eca4c215bc91138a"
server_key_version: "encryptionkey@0"
I20260926 23:20:09.760880 32459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:09.773665 32459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:09.777732 32459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:09.778035 32459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:09.778962 32459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:20:09.780467 32459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:20:09.780593 32459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:09.780736 32459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:20:09.780822 32459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:09.815129 32459 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:40889
I20260926 23:20:09.815177 32586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:40889 every 8 connection(s)
I20260926 23:20:09.816509 32459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:20:09.817826 32320 heartbeater.cc:503] Master 127.16.62.126:39965 was elected leader, sending a full tablet report...
I20260926 23:20:09.821048 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 32459
I20260926 23:20:09.821254 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:20:09.821708 16633 external_mini_cluster.cc:1428] Setting key 390741dc18d08c283cd1bf34910daa3e
Sep 26 23:20:09 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464809, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:20:09.836786 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.819249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:48359 (local address 127.16.62.126:39965)
0926 23:20:09.819705 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:09.819711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:09.820824 (+ 1113us) server_negotiation.cc:407] Connection header received
0926 23:20:09.822890 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:09.822896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:09.822976 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:09.823104 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:09.824641 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.825403 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.826930 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.827364 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.827498 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.831044 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:09.831049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:09.831052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:09.831084 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:09.833669 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:09.834327 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:09.834330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:09.834332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:09.834441 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:09.835008 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:09.835012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:09.835016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:09.835318 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:09.835487 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:09.836203 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 23:20:09.836422 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":359,"threads_started":1}
I20260926 23:20:09.837723 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.819941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:48359)
0926 23:20:09.820684 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:09.820720 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:09.822627 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:09.823341 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:09.823349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:09.823847 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:20:09.824470 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.824487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.825565 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.825575 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.826758 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.826769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.827613 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.827620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.827911 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.828609 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:20:09.828631 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:20:09.830820 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:20:09.833793 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:09.833799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:09.833810 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:09.834219 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:09.834561 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:09.834564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:09.834566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:09.834816 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:09.835459 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:20:09.835466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:20:09.835805 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 23:20:09.836068 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 23:20:09.836605 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":316,"threads_started":1}
I20260926 23:20:09.838927 32587 heartbeater.cc:347] Connected to a master server at 127.16.62.126:39965
I20260926 23:20:09.839226 32587 heartbeater.cc:464] Registering TS with master...
I20260926 23:20:09.839907 32587 heartbeater.cc:511] Master 127.16.62.126:39965 requested a full tablet report, sending...
I20260926 23:20:09.841285 32131 ts_manager.cc:194] Registered new tserver with Master: 787e53d925364e7581f8068528df71bd (127.16.62.67:40889)
I20260926 23:20:09.842047 32131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:48359
I20260926 23:20:09.848714 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:20:09.865664 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.853097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.16.62.126:39965)
0926 23:20:09.853268 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:09.853273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:09.853746 (+ 473us) server_negotiation.cc:407] Connection header received
0926 23:20:09.853935 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:09.853940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:09.854007 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:09.854112 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:09.855531 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.856318 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.857687 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.858089 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.858223 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.860685 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:09.860697 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:09.860702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:09.860732 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:09.863062 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:09.863724 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:09.863731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:09.863734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:09.863859 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:09.864330 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:09.864336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:09.864339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:09.864675 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:09.864928 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:09.865265 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 23:20:09.865428 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:20:09.869773 32131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:20:09.876617 32131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:20:09.910939 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.888732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:40889 (local address 127.0.0.1:43036)
0926 23:20:09.889691 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:09.889735 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:09.889934 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:09.902975 (+ 13041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:09.902982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:09.903015 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:09.903483 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.903497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.905432 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.905440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.908339 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.908355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.910078 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.910087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.910245 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.910275 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:20:09.910377 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 23:20:09.910517 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":205,"thread_start_us":549,"threads_started":1}
I20260926 23:20:09.911408 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.890049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.16.62.65:35069)
0926 23:20:09.902613 (+ 12564us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:09.902620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:09.902644 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:20:09.902722 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:09.902728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:09.902917 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:09.903454 (+ 537us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:09.903903 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.905649 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.908531 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.909471 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.909638 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.909889 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:09.910932 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0926 23:20:09.911113 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12375,"thread_start_us":12433,"threads_started":1}
I20260926 23:20:09.911495 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.889149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.16.62.67:40889)
0926 23:20:09.902271 (+ 13122us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:09.902284 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:09.902324 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:20:09.902433 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:09.902444 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:09.902734 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:09.902967 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:09.903911 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.905307 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.908504 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.909875 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.910013 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.910230 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:09.911047 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 23:20:09.911211 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12949,"thread_start_us":12780,"threads_started":1}
I20260926 23:20:09.911211 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.890832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.16.62.66:34229)
0926 23:20:09.901725 (+ 10893us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:09.901738 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:09.902389 (+ 651us) server_negotiation.cc:407] Connection header received
0926 23:20:09.902538 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:09.902546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:09.902790 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:09.903005 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:09.904528 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.905983 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.908425 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.909875 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.910013 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.910196 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:09.910721 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 23:20:09.910889 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10664,"thread_start_us":10494,"threads_started":1}
I20260926 23:20:09.910791 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.889425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35069 (local address 127.0.0.1:33348)
0926 23:20:09.900119 (+ 10694us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:09.900162 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:09.900354 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:09.903131 (+ 2777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:09.903136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:09.903172 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:09.903667 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.903681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.905788 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.905793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.908372 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.908382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.909683 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.909689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.909886 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.909949 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 23:20:09.910106 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 23:20:09.910289 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10495,"mutex_wait_us":158,"thread_start_us":11049,"threads_started":1}
I20260926 23:20:09.911981 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.890253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:34229 (local address 127.0.0.1:56606)
0926 23:20:09.902241 (+ 11988us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:09.902261 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:09.902400 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:09.903294 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:09.903299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:09.903325 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:09.904339 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.904355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.906126 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.906132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.908266 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.908279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.910395 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.910400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.910519 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.910542 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:20:09.910667 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 23:20:09.910804 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11872,"mutex_wait_us":39,"thread_start_us":11727,"threads_started":1}
I20260926 23:20:09.915340 32522 tablet_service.cc:1467] Processing CreateTablet for tablet aa3effe59dd34067a97a8a7d1a53da48 (DEFAULT_TABLE table=test-table [id=36d2494cd18d425ba6aefdada4bb2bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:20:09.915571 32255 tablet_service.cc:1467] Processing CreateTablet for tablet aa3effe59dd34067a97a8a7d1a53da48 (DEFAULT_TABLE table=test-table [id=36d2494cd18d425ba6aefdada4bb2bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:20:09.916865 32522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3effe59dd34067a97a8a7d1a53da48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:20:09.916440 32389 tablet_service.cc:1467] Processing CreateTablet for tablet aa3effe59dd34067a97a8a7d1a53da48 (DEFAULT_TABLE table=test-table [id=36d2494cd18d425ba6aefdada4bb2bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:20:09.917064 32255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3effe59dd34067a97a8a7d1a53da48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:20:09.917747 32389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3effe59dd34067a97a8a7d1a53da48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:20:09.928933 32606 tablet_bootstrap.cc:492] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: Bootstrap starting.
I20260926 23:20:09.931318 32607 tablet_bootstrap.cc:492] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: Bootstrap starting.
I20260926 23:20:09.932926 32608 tablet_bootstrap.cc:492] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: Bootstrap starting.
I20260926 23:20:09.932991 32606 tablet_bootstrap.cc:654] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: Neither blocks nor log segments found. Creating new log.
I20260926 23:20:09.934309 32606 log.cc:824] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: Log is configured to *not* fsync() on all Append() calls
I20260926 23:20:09.934783 32607 tablet_bootstrap.cc:654] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: Neither blocks nor log segments found. Creating new log.
I20260926 23:20:09.936069 32608 tablet_bootstrap.cc:654] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: Neither blocks nor log segments found. Creating new log.
I20260926 23:20:09.936038 32607 log.cc:824] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: Log is configured to *not* fsync() on all Append() calls
I20260926 23:20:09.937255 32608 log.cc:824] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: Log is configured to *not* fsync() on all Append() calls
I20260926 23:20:09.937587 32606 tablet_bootstrap.cc:492] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: No bootstrap required, opened a new log
I20260926 23:20:09.937863 32606 ts_tablet_manager.cc:1397] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 23:20:09.939278 32607 tablet_bootstrap.cc:492] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: No bootstrap required, opened a new log
I20260926 23:20:09.939553 32607 ts_tablet_manager.cc:1397] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 23:20:09.940356 32608 tablet_bootstrap.cc:492] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: No bootstrap required, opened a new log
I20260926 23:20:09.940605 32608 ts_tablet_manager.cc:1397] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 23:20:09.947252 32608 raft_consensus.cc:348] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:09.947628 32608 raft_consensus.cc:374] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:20:09.947736 32608 raft_consensus.cc:729] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcc77497d6b49ef98c2242add6b8170, State: Initialized, Role: FOLLOWER
I20260926 23:20:09.948380 32608 consensus_queue.cc:260] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:09.948589 32606 raft_consensus.cc:348] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:09.949054 32606 raft_consensus.cc:374] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:20:09.949196 32606 raft_consensus.cc:729] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787e53d925364e7581f8068528df71bd, State: Initialized, Role: FOLLOWER
I20260926 23:20:09.949908 32606 consensus_queue.cc:260] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:09.950071 32607 raft_consensus.cc:348] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:09.950573 32607 raft_consensus.cc:374] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:20:09.950733 32607 raft_consensus.cc:729] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f8a7f5c5614bd5b3bc925cc52411a0, State: Initialized, Role: FOLLOWER
I20260926 23:20:09.950925 32454 heartbeater.cc:503] Master 127.16.62.126:39965 was elected leader, sending a full tablet report...
I20260926 23:20:09.951628 32608 ts_tablet_manager.cc:1428] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 23:20:09.951443 32607 consensus_queue.cc:260] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:09.953567 32607 ts_tablet_manager.cc:1428] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260926 23:20:09.956089 32606 ts_tablet_manager.cc:1428] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260926 23:20:09.956274 32587 heartbeater.cc:503] Master 127.16.62.126:39965 was elected leader, sending a full tablet report...
I20260926 23:20:09.967595 32612 raft_consensus.cc:482] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:20:09.967805 32612 raft_consensus.cc:504] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:09.969357 32612 leader_election.cc:288] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 787e53d925364e7581f8068528df71bd (127.16.62.67:40889), e0f8a7f5c5614bd5b3bc925cc52411a0 (127.16.62.65:35069)
I20260926 23:20:09.978513 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.969745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:55523 (local address 127.16.62.67:40889)
0926 23:20:09.969959 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:09.969964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:09.970730 (+ 766us) server_negotiation.cc:407] Connection header received
0926 23:20:09.970932 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:09.970937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:09.971011 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:09.971106 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:09.971924 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.973067 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.976251 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.977458 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.977636 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.977701 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:09.978132 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 23:20:09.978274 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:20:09.978699 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.969756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:49139 (local address 127.16.62.65:35069)
0926 23:20:09.969958 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:09.969965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:09.970730 (+ 765us) server_negotiation.cc:407] Connection header received
0926 23:20:09.970933 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:09.970938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:09.971011 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:09.971119 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:09.971956 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.973061 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.976261 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.977463 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.977629 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.977701 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:09.978340 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 23:20:09.978471 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 23:20:09.978700 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.969685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:40889 (local address 127.16.62.66:55523)
0926 23:20:09.970604 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:09.970628 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:09.970810 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:09.971139 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:09.971144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:09.971190 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:09.971757 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.971775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.973221 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.973226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.976080 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.976094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.977722 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.977729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.977925 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.977961 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:20:09.978077 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 23:20:09.978213 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":199,"spinlock_wait_cycles":37888,"thread_start_us":495,"threads_started":1}
I20260926 23:20:09.979437 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:09.969716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35069 (local address 127.16.62.66:49139)
0926 23:20:09.970627 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:09.970642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:09.970769 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:09.971139 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:09.971144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:09.971188 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:09.971817 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.971830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.973221 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.973226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.976082 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:09.976094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:09.977934 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:09.977940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:09.978164 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:09.978195 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:20:09.978982 (+ 787us) client_negotiation.cc:240] Negotiation successful
0926 23:20:09.979177 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":351,"spinlock_wait_cycles":13312,"thread_start_us":496,"threads_started":1}
I20260926 23:20:09.979692 32542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3effe59dd34067a97a8a7d1a53da48" candidate_uuid: "5bcc77497d6b49ef98c2242add6b8170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787e53d925364e7581f8068528df71bd" is_pre_election: true
I20260926 23:20:09.980304 32542 raft_consensus.cc:2454] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bcc77497d6b49ef98c2242add6b8170 in term 0.
I20260926 23:20:09.980496 32275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3effe59dd34067a97a8a7d1a53da48" candidate_uuid: "5bcc77497d6b49ef98c2242add6b8170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" is_pre_election: true
I20260926 23:20:09.980990 32275 raft_consensus.cc:2454] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bcc77497d6b49ef98c2242add6b8170 in term 0.
I20260926 23:20:09.981002 32342 leader_election.cc:302] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bcc77497d6b49ef98c2242add6b8170, 787e53d925364e7581f8068528df71bd; no voters:
I20260926 23:20:09.981489 32612 raft_consensus.cc:2781] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:20:09.981685 32612 raft_consensus.cc:482] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:20:09.981832 32612 raft_consensus.cc:3037] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:20:09.984323 32612 raft_consensus.cc:504] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:09.984956 32612 leader_election.cc:288] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [CANDIDATE]: Term 1 election: Requested vote from peers 787e53d925364e7581f8068528df71bd (127.16.62.67:40889), e0f8a7f5c5614bd5b3bc925cc52411a0 (127.16.62.65:35069)
I20260926 23:20:09.985378 32542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3effe59dd34067a97a8a7d1a53da48" candidate_uuid: "5bcc77497d6b49ef98c2242add6b8170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787e53d925364e7581f8068528df71bd"
I20260926 23:20:09.985391 32275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3effe59dd34067a97a8a7d1a53da48" candidate_uuid: "5bcc77497d6b49ef98c2242add6b8170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0"
I20260926 23:20:09.985587 32275 raft_consensus.cc:3037] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:20:09.985582 32542 raft_consensus.cc:3037] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:20:09.987699 32275 raft_consensus.cc:2454] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bcc77497d6b49ef98c2242add6b8170 in term 1.
I20260926 23:20:09.987912 32542 raft_consensus.cc:2454] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bcc77497d6b49ef98c2242add6b8170 in term 1.
I20260926 23:20:09.988132 32343 leader_election.cc:302] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bcc77497d6b49ef98c2242add6b8170, e0f8a7f5c5614bd5b3bc925cc52411a0; no voters:
I20260926 23:20:09.988497 32612 raft_consensus.cc:2781] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:20:09.989017 32612 raft_consensus.cc:686] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 LEADER]: Becoming Leader. State: Replica: 5bcc77497d6b49ef98c2242add6b8170, State: Running, Role: LEADER
I20260926 23:20:09.989547 32612 consensus_queue.cc:237] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:09.994200 32131 catalog_manager.cc:5495] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bcc77497d6b49ef98c2242add6b8170 (127.16.62.66). New cstate: current_term: 1 leader_uuid: "5bcc77497d6b49ef98c2242add6b8170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } health_report { overall_health: UNKNOWN } } }
I20260926 23:20:10.010272 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:10.004691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.16.62.66:34229)
0926 23:20:10.004902 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:10.004908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:10.004981 (+ 73us) server_negotiation.cc:407] Connection header received
0926 23:20:10.005253 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:10.005257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:10.005331 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:10.005487 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:20:10.005882 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:10.006614 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:10.008170 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:10.008594 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:10.008727 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:10.009215 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:20:10.009685 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:20:10.009841 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:10.009920 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 23:20:10.009999 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:20:10.018304 32542 raft_consensus.cc:1261] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [term 1 FOLLOWER]: Refusing update from remote peer 5bcc77497d6b49ef98c2242add6b8170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:20:10.018465 32275 raft_consensus.cc:1261] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 1 FOLLOWER]: Refusing update from remote peer 5bcc77497d6b49ef98c2242add6b8170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:20:10.019332 32617 consensus_queue.cc:1035] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:20:10.019729 32612 consensus_queue.cc:1035] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:20:10.031322 32620 mvcc.cc:205] Tried to move back new op lower bound from 7333743861824593920 to 7333743861721505792. Current Snapshot: MvccSnapshot[applied={T|T < 7333743861824593920}]
I20260926 23:20:10.046011 32131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:20:10.046257 32131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:20:10.049976 32131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Sending DeleteTablet for 3 replicas of tablet aa3effe59dd34067a97a8a7d1a53da48
I20260926 23:20:10.051168 32255 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3effe59dd34067a97a8a7d1a53da48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:20:10 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:33348
I20260926 23:20:10.051198 32389 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3effe59dd34067a97a8a7d1a53da48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:20:10 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:56606
I20260926 23:20:10.051326 32522 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3effe59dd34067a97a8a7d1a53da48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:20:10 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:43036
I20260926 23:20:10.052551 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 32192
I20260926 23:20:10.061786 32630 tablet_replica.cc:330] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: stopping tablet replica
I20260926 23:20:10.062359 32630 raft_consensus.cc:2229] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:20:10.063009 32630 raft_consensus.cc:2258] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:20:10.065438 32630 ts_tablet_manager.cc:1905] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:20:10.072635 32632 tablet_replica.cc:330] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: stopping tablet replica
I20260926 23:20:10.072950 32631 tablet_replica.cc:330] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: stopping tablet replica
I20260926 23:20:10.073113 32632 raft_consensus.cc:2229] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:20:10.073436 32631 raft_consensus.cc:2229] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:20:10.073647 32632 raft_consensus.cc:2258] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:20:10.074209 32631 raft_consensus.cc:2258] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:20:10.075464 32630 ts_tablet_manager.cc:1918] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:20:10.076215 32632 ts_tablet_manager.cc:1905] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:20:10.076756 32631 ts_tablet_manager.cc:1905] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:20:10.077049 32630 log.cc:1196] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/aa3effe59dd34067a97a8a7d1a53da48
I20260926 23:20:10.077634 32630 ts_tablet_manager.cc:1939] T aa3effe59dd34067a97a8a7d1a53da48 P e0f8a7f5c5614bd5b3bc925cc52411a0: Deleting consensus metadata
I20260926 23:20:10.083282 32632 ts_tablet_manager.cc:1918] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:20:10.083551 32632 log.cc:1196] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/aa3effe59dd34067a97a8a7d1a53da48
Sep 26 23:20:10 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:20:10.084237 32632 ts_tablet_manager.cc:1939] T aa3effe59dd34067a97a8a7d1a53da48 P 787e53d925364e7581f8068528df71bd: Deleting consensus metadata
I20260926 23:20:10.086297 32115 catalog_manager.cc:4841] TS 787e53d925364e7581f8068528df71bd (127.16.62.67:40889): tablet aa3effe59dd34067a97a8a7d1a53da48 (table test-table [id=36d2494cd18d425ba6aefdada4bb2bff]) successfully deleted
I20260926 23:20:10.086354 32116 catalog_manager.cc:4841] TS e0f8a7f5c5614bd5b3bc925cc52411a0 (127.16.62.65:35069): tablet aa3effe59dd34067a97a8a7d1a53da48 (table test-table [id=36d2494cd18d425ba6aefdada4bb2bff]) successfully deleted
I20260926 23:20:10.087332 32631 ts_tablet_manager.cc:1918] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:20:10.087528 32631 log.cc:1196] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/aa3effe59dd34067a97a8a7d1a53da48
I20260926 23:20:10.088074 32631 ts_tablet_manager.cc:1939] T aa3effe59dd34067a97a8a7d1a53da48 P 5bcc77497d6b49ef98c2242add6b8170: Deleting consensus metadata
I20260926 23:20:10.089654 32115 catalog_manager.cc:4841] TS 5bcc77497d6b49ef98c2242add6b8170 (127.16.62.66:34229): tablet aa3effe59dd34067a97a8a7d1a53da48 (table test-table [id=36d2494cd18d425ba6aefdada4bb2bff]) successfully deleted
I20260926 23:20:10.091681 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:10.053808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.16.62.65:35069)
0926 23:20:10.054027 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:10.054037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:10.070466 (+ 16429us) server_negotiation.cc:407] Connection header received
0926 23:20:10.073036 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:10.073044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:10.073148 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:10.073223 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:10.075896 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:10.076985 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:10.078992 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:10.079639 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:10.079744 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:10.085424 (+ 5680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:10.085440 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:10.085459 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:10.085507 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:10.088820 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:10.089614 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:10.089623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:10.089627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:10.089760 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:10.090141 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:10.090145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:10.090148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:10.090476 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:10.090796 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:10.091101 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 23:20:10.091315 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 23:20:10.092383 32315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:10.182049 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 32192
I20260926 23:20:10.206118 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 32326
Sep 26 23:20:10 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:20:10.222785 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:10.207060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.16.62.66:34229)
0926 23:20:10.207260 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:10.207265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:10.207286 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:20:10.207400 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:10.207406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:10.207489 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:10.207603 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:10.209054 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:10.209885 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:10.211305 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:10.211755 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:10.211899 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:10.216576 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:10.216590 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:10.216605 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:10.216647 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:10.219881 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:10.220650 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:10.220658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:10.220660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:10.220797 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:10.221216 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:10.221225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:10.221227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:10.221577 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:10.221901 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:10.222209 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:20:10.222427 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:20:10.223532 32449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:10.334694 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 32326
I20260926 23:20:10.353236 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 32459
Sep 26 23:20:10 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:20:10.368671 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:10.354058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.16.62.67:40889)
0926 23:20:10.354251 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:10.354257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:10.354277 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:20:10.354373 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:10.354379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:10.354459 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:10.354575 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:10.355919 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:10.356730 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:10.358060 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:10.358442 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:10.358576 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:10.362965 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:10.362984 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:10.363000 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:10.363031 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:10.365951 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:10.366618 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:10.366631 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:10.366633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:10.366763 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:10.367143 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:10.367150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:10.367153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:10.367481 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:10.367784 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:10.368075 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 23:20:10.368377 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:20:10.369279 32582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:10.459162 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 32459
I20260926 23:20:10.476536 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 32099
I20260926 23:20:10.477663 32161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:10.587664 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 32099
I20260926 23:20:30.609066 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:39965
--webserver_interface=127.16.62.126
--webserver_port=37721
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:39965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:20:30.933430 32638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:30.933693 32638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:30.933861 32638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:30.940910 32638 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 23:20:30.941025 32638 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 23:20:30.941099 32638 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 23:20:30.941184 32638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:20:30.941270 32638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:30.941387 32638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:30.941449 32638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:20:30.941515 32638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:20:30.951138 32638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:39965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:39965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=37721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:30.952374 32638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:30.953649 32638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:30.975684 32645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:30.975780 32647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:30.975740 32644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:30.976605 32638 server_base.cc:1034] running on GCE node
I20260926 23:20:30.977465 32638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:30.979048 32638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:30.980284 32638 hybrid_clock.cc:648] HybridClock initialized: now 1790464830980222 us; error 36 us; skew 500 ppm
I20260926 23:20:30.980685 32638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:30 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464830, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:30.984205 32638 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:20:30.985903 32638 webserver.cc:460] Webserver started at http://127.16.62.126:37721/ using document root <none> and password file <none>
I20260926 23:20:30.986665 32638 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:30.986807 32638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:30.992705 32638 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:20:30.995523 32654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:30.996925 32638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:20:30.997112 32638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "03c42eba704743a580e11a213a3a26a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "40de4a8c403b3a97b16fc6504f0ed0cb"
server_key_iv: "8d6117d72cc9d5c2181bda4221c5114e"
server_key_version: "encryptionkey@0"
I20260926 23:20:30.998149 32638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:31.036103 32638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:31.040233 32638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:31.040530 32638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:31.060148 32638 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:39965
I20260926 23:20:31.060247 32705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:39965 every 8 connection(s)
I20260926 23:20:31.061689 32638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:20:31.068871 32706 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:20:31.071933 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 32638
I20260926 23:20:31.072276 32706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Bootstrap starting.
I20260926 23:20:31.072844 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:35069
--local_ip_for_outbound_sockets=127.16.62.65
--tserver_master_addrs=127.16.62.126:39965
--webserver_port=36987
--webserver_interface=127.16.62.65
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:20:31.092724 32706 log.cc:824] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Log is configured to *not* fsync() on all Append() calls
I20260926 23:20:31.106106 32706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:20:31.106670 32706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Bootstrap complete.
I20260926 23:20:31.116276 32706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:31.117175 32706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c42eba704743a580e11a213a3a26a0, State: Initialized, Role: FOLLOWER
I20260926 23:20:31.117842 32706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:31.118116 32706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:20:31.118252 32706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:20:31.118441 32706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:20:31.122400 32706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:31.122941 32706 leader_election.cc:302] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03c42eba704743a580e11a213a3a26a0; no voters:
I20260926 23:20:31.123689 32706 leader_election.cc:288] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:20:31.123828 32710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:20:31.125229 32710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 2 LEADER]: Becoming Leader. State: Replica: 03c42eba704743a580e11a213a3a26a0, State: Running, Role: LEADER
I20260926 23:20:31.125917 32710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:31.126214 32706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:20:31.128815 32712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03c42eba704743a580e11a213a3a26a0. Latest consensus state: current_term: 2 leader_uuid: "03c42eba704743a580e11a213a3a26a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } } }
I20260926 23:20:31.129230 32712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:20:31.129146 32711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "03c42eba704743a580e11a213a3a26a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } } }
I20260926 23:20:31.129829 32711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:20:31.142390 32716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:20:31.150885 32716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:20:31.152880 32716 catalog_manager.cc:1259] Loaded cluster ID: 450da93f999a45659d7f54aa4f962977
I20260926 23:20:31.153101 32716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:20:31.158453 32716 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:20:31.160023 32716 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Loaded TSK: 0
I20260926 23:20:31.196066 32716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Generated new TSK 1
I20260926 23:20:31.196519 32716 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:20:31.408687 32708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:31.408978 32708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:31.409135 32708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:31.416172 32708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:31.416321 32708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:31.416492 32708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:20:31.426126 32708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:35069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=36987
--tserver_master_addrs=127.16.62.126:39965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:31.427495 32708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:31.428882 32708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:31.443058 32733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:31.443079 32734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:31.444178 32736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:31.444487 32708 server_base.cc:1034] running on GCE node
I20260926 23:20:31.445144 32708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:31.446153 32708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:31.447368 32708 hybrid_clock.cc:648] HybridClock initialized: now 1790464831447325 us; error 38 us; skew 500 ppm
I20260926 23:20:31.447687 32708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:31 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:31.451181 32708 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:20:31.452814 32708 webserver.cc:460] Webserver started at http://127.16.62.65:36987/ using document root <none> and password file <none>
I20260926 23:20:31.453495 32708 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:31.453646 32708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:31.459072 32708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:20:31.461882 32743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:31.463563 32708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:20:31.463742 32708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "0e3b17b3f1591a63c914485a24ad28ec"
server_key_iv: "42eed758b86e764845e8e688f51280e7"
server_key_version: "encryptionkey@0"
I20260926 23:20:31.464950 32708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:31.478031 32708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:31.482013 32708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:31.482281 32708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:31.483361 32708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:20:31.484766 32708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:20:31.484867 32708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:31.484983 32708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:20:31.485072 32708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:31.521540 32708 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:35069
I20260926 23:20:31.521625 387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:35069 every 8 connection(s)
I20260926 23:20:31.522974 32708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:20:31.526005 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 32708
I20260926 23:20:31.526700 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:34229
--local_ip_for_outbound_sockets=127.16.62.66
--tserver_master_addrs=127.16.62.126:39965
--webserver_port=42315
--webserver_interface=127.16.62.66
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:20:31 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:20:31.548110 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:31.525992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:42483)
0926 23:20:31.526897 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:31.526949 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:31.528784 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:31.530332 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:31.530339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:31.530840 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:20:31.531360 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:31.531376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:31.533708 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:31.533712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:31.534893 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:31.534904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:31.535960 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:31.535967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:31.536264 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:31.536927 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:20:31.536948 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:20:31.539454 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:20:31.544047 (+ 4593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:31.544058 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:31.544071 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:31.544559 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:31.545005 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:31.545010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:31.545014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:31.545197 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:31.546259 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:20:31.546265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:20:31.546533 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 23:20:31.546757 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 23:20:31.547138 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":368,"threads_started":1}
I20260926 23:20:31.548944 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:31.525707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:42483 (local address 127.16.62.126:39965)
0926 23:20:31.526378 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:31.526397 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:31.527108 (+ 711us) server_negotiation.cc:407] Connection header received
0926 23:20:31.529360 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:31.529381 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:31.529824 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:31.530231 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:31.531558 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:31.533552 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:31.535052 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:31.535718 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:31.535930 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:31.539721 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:31.539729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:31.539743 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:31.539777 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:31.543868 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:31.544699 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:31.544708 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:31.544715 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:31.544885 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:31.545370 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:31.545377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:31.545383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:31.546114 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:31.546353 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:31.547054 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 23:20:31.547546 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":251,"threads_started":1}
I20260926 23:20:31.551520 388 heartbeater.cc:347] Connected to a master server at 127.16.62.126:39965
I20260926 23:20:31.551818 388 heartbeater.cc:464] Registering TS with master...
I20260926 23:20:31.552542 388 heartbeater.cc:511] Master 127.16.62.126:39965 requested a full tablet report, sending...
I20260926 23:20:31.554589 32671 ts_manager.cc:194] Registered new tserver with Master: e0f8a7f5c5614bd5b3bc925cc52411a0 (127.16.62.65:35069)
I20260926 23:20:31.556229 32671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:42483
W20260926 23:20:31.852367 392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:31.852674 392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:31.852830 392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:31.859850 392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:31.859984 392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:31.860212 392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:20:31.869586 392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:34229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=42315
--tserver_master_addrs=127.16.62.126:39965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:31.870811 392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:31.872097 392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:31.884656 400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:31.884857 397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:31.884691 398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:31.885689 392 server_base.cc:1034] running on GCE node
I20260926 23:20:31.886260 392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:31.887209 392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:31.888412 392 hybrid_clock.cc:648] HybridClock initialized: now 1790464831888369 us; error 38 us; skew 500 ppm
I20260926 23:20:31.888707 392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:31 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:31.891774 392 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:20:31.893177 392 webserver.cc:460] Webserver started at http://127.16.62.66:42315/ using document root <none> and password file <none>
I20260926 23:20:31.893838 392 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:31.893985 392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:31.899223 392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 23:20:31.901871 408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:31.903242 392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:20:31.903407 392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "5bcc77497d6b49ef98c2242add6b8170"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "e5500f1db782a9cf8c5652145f415117"
server_key_iv: "33ca35f637fbc692eddf0bf3130d58de"
server_key_version: "encryptionkey@0"
I20260926 23:20:31.904372 392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:31.922221 392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:31.925872 392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:31.926168 392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:31.927232 392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:20:31.928638 392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:20:31.928753 392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:31.928872 392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:20:31.928954 392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:31.960863 392 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:34229
I20260926 23:20:31.960915 522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:34229 every 8 connection(s)
I20260926 23:20:31.962471 392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:20:31.971698 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 392
I20260926 23:20:31.972357 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:40889
--local_ip_for_outbound_sockets=127.16.62.67
--tserver_master_addrs=127.16.62.126:39965
--webserver_port=33709
--webserver_interface=127.16.62.67
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:20:31 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:20:31.982132 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:31.964965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:41147 (local address 127.16.62.126:39965)
0926 23:20:31.965148 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:31.965152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:31.966245 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 23:20:31.967982 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:31.967989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:31.968076 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:31.968245 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:31.969834 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:31.970518 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:31.972067 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:31.972628 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:31.972768 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:31.976248 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:31.976254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:31.976259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:31.976301 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:31.979198 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:31.979880 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:31.979888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:31.979893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:31.980016 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:31.980466 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:31.980473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:31.980476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:31.980758 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:31.980901 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:31.981554 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 23:20:31.981796 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:20:31.983080 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:31.965421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:41147)
0926 23:20:31.966095 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:31.966131 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:31.967718 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:31.968624 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:31.968632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:31.969108 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:20:31.969680 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:31.969697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:31.970671 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:31.970677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:31.971937 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:31.971948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:31.972861 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:31.972868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:31.973090 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:31.973738 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:20:31.973764 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:20:31.976022 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:20:31.979334 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:31.979343 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:31.979359 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:31.979766 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:31.980128 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:31.980133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:31.980173 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:31.980358 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:31.980885 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:20:31.980893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:20:31.981221 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 23:20:31.981479 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 23:20:31.981983 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":262,"threads_started":1}
I20260926 23:20:31.984607 523 heartbeater.cc:347] Connected to a master server at 127.16.62.126:39965
I20260926 23:20:31.984910 523 heartbeater.cc:464] Registering TS with master...
I20260926 23:20:31.985599 523 heartbeater.cc:511] Master 127.16.62.126:39965 requested a full tablet report, sending...
I20260926 23:20:31.987035 32671 ts_manager.cc:194] Registered new tserver with Master: 5bcc77497d6b49ef98c2242add6b8170 (127.16.62.66:34229)
I20260926 23:20:31.987957 32671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:41147
W20260926 23:20:32.307955 526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:32.308377 526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:32.308557 526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:32.316092 526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:32.316263 526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:32.316444 526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:20:32.326623 526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:40889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=33709
--tserver_master_addrs=127.16.62.126:39965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:32.327986 526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:32.329321 526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:32.343132 531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:32.343351 534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:32.343645 532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:32.344072 526 server_base.cc:1034] running on GCE node
I20260926 23:20:32.344807 526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:32.345803 526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:32.347033 526 hybrid_clock.cc:648] HybridClock initialized: now 1790464832346986 us; error 37 us; skew 500 ppm
I20260926 23:20:32.347388 526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:32 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:32.350620 526 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:20:32.352197 526 webserver.cc:460] Webserver started at http://127.16.62.67:33709/ using document root <none> and password file <none>
I20260926 23:20:32.352875 526 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:32.353024 526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:32.358345 526 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:20:32.360941 541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:32.362196 526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:20:32.362363 526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "787e53d925364e7581f8068528df71bd"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "132d6bf632faa60216fb951ebb278014"
server_key_iv: "0dff8567f3e613f8eca4c215bc91138a"
server_key_version: "encryptionkey@0"
I20260926 23:20:32.363314 526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:32.390282 526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:32.395022 526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:32.395264 526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:32.396301 526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:20:32.397634 526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:20:32.397719 526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:32.397799 526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:20:32.397852 526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:32.430120 526 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:40889
I20260926 23:20:32.430214 654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:40889 every 8 connection(s)
I20260926 23:20:32.431565 526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:20:32.437815 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 526
Sep 26 23:20:32 dist-test-slave-vcv1 krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:20:32.454509 659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.438736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.16.62.126:39965)
0926 23:20:32.439555 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.439559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.439573 (+ 14us) server_negotiation.cc:407] Connection header received
0926 23:20:32.439614 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.439617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.439665 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.439759 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:32.441743 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.442877 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.444658 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.445071 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.445199 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.448533 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:32.448539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:32.448542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:32.448581 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:32.451578 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:32.452411 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:32.452418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:32.452424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:32.452553 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:32.452958 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:32.452964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:32.452968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:32.453281 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:32.453532 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:32.453826 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 23:20:32.454080 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":735,"thread_start_us":605,"threads_started":1}
I20260926 23:20:32.454604 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.434252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:51269 (local address 127.16.62.126:39965)
0926 23:20:32.434453 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.434460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.435587 (+ 1127us) server_negotiation.cc:407] Connection header received
0926 23:20:32.437630 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.437636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.437727 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.437833 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:32.439638 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.440914 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.443036 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.443716 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.443857 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.447478 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:32.447487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:32.447492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:32.447538 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:32.450892 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:32.451840 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:32.451846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:32.451850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:32.451984 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:32.452462 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:32.452465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:32.452467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:32.452837 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:32.452987 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:32.453877 (+ 890us) server_negotiation.cc:299] Negotiation successful
0926 23:20:32.454226 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":71}
I20260926 23:20:32.455596 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.434701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:51269)
0926 23:20:32.435412 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:32.435450 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:32.437334 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:32.438084 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:32.438092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:32.438682 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:20:32.439454 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.439481 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.441083 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.441087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.442866 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.442880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.443914 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.443919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.444110 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.444826 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:20:32.444848 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:20:32.447255 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:20:32.451025 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:32.451035 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:32.451055 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:32.451723 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:32.452095 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:20:32.452098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:20:32.452100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:20:32.452354 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:20:32.452977 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:20:32.452984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:20:32.453431 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 23:20:32.453767 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 23:20:32.454315 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":279,"threads_started":1}
I20260926 23:20:32.457108 655 heartbeater.cc:347] Connected to a master server at 127.16.62.126:39965
I20260926 23:20:32.457427 655 heartbeater.cc:464] Registering TS with master...
I20260926 23:20:32.458127 655 heartbeater.cc:511] Master 127.16.62.126:39965 requested a full tablet report, sending...
I20260926 23:20:32.459488 32671 ts_manager.cc:194] Registered new tserver with Master: 787e53d925364e7581f8068528df71bd (127.16.62.67:40889)
I20260926 23:20:32.460557 32671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:51269
I20260926 23:20:32.469944 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:20:32.476464 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.470897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.16.62.126:39965)
0926 23:20:32.471055 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.471059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.471563 (+ 504us) server_negotiation.cc:407] Connection header received
0926 23:20:32.471723 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.471727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.471872 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.471996 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:20:32.472460 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.473224 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.474657 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.475056 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.475149 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.475579 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:20:32.475875 (+ 296us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 23:20:32.476276 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54672: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 23:20:32.476567 390 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54672: token expired
W20260926 23:20:32.477152 32594 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.62.126:39965)
I20260926 23:20:32.489966 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.477675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.16.62.126:39965)
0926 23:20:32.477814 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.477818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.477831 (+ 13us) server_negotiation.cc:407] Connection header received
0926 23:20:32.477866 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.477869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.477921 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.477994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:32.479324 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.480064 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.481476 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.481868 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.481964 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.484628 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:32.484633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:32.484637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:32.484672 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:32.487196 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:32.487905 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:32.487910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:32.487912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:32.488053 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:32.488590 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:32.488594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:32.488597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:32.488956 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:32.489155 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:32.489503 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 23:20:32.489687 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 23:20:32.493643 32671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:20:32.501447 32671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:20:32.529223 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.513523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:34229 (local address 127.0.0.1:44888)
0926 23:20:32.514290 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:32.514337 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:32.514527 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:32.515738 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:32.515745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:32.515779 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:32.516247 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.516262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.519002 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.519011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.522354 (+ 3343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.522375 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.528033 (+ 5658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.528039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.528303 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.528378 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 23:20:32.528481 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 23:20:32.528704 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":1,"thread_start_us":346,"threads_started":1}
I20260926 23:20:32.530483 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.513746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.16.62.66:34229)
0926 23:20:32.515072 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.515083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.515112 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:20:32.515222 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.515229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.515468 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.516492 (+ 1024us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:32.516726 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.518801 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.524695 (+ 5894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.527792 (+ 3097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.529717 (+ 1925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.529921 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:32.529998 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 23:20:32.530160 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1161,"thread_start_us":1040,"threads_started":1}
I20260926 23:20:32.534050 457 tablet_service.cc:1467] Processing CreateTablet for tablet f81e62dda14046d5b2efcf23fdc7d1d3 (DEFAULT_TABLE table=test-table [id=268c646c111748138bc8082ba410d597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:20:32.534921 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.515286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35069 (local address 127.0.0.1:50430)
0926 23:20:32.522602 (+ 7316us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:32.522620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:32.522724 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:32.526249 (+ 3525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:32.526254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:32.526280 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:32.526750 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.526765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.528938 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.528944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.532449 (+ 3505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.532466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.534100 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.534106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.534268 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.534308 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 23:20:32.534418 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 23:20:32.534547 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7179,"mutex_wait_us":79,"thread_start_us":6990,"threads_started":1}
I20260926 23:20:32.534937 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.515802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.16.62.65:35069)
0926 23:20:32.524717 (+ 8915us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.524730 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.524768 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:20:32.525061 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.525072 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.525382 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.525627 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:32.526958 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.528766 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.532630 (+ 3864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.533880 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.534033 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.534271 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:32.534462 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 23:20:32.534617 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8696,"thread_start_us":8548,"threads_started":1}
I20260926 23:20:32.536001 457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f81e62dda14046d5b2efcf23fdc7d1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:20:32.536128 667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.514832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:40889 (local address 127.0.0.1:55048)
0926 23:20:32.520108 (+ 5276us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:32.520153 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:32.520305 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:32.528451 (+ 8146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:32.528458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:32.528486 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:32.528849 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.528862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.531645 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.531648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.533363 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.533377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.535147 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.535153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.535320 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.535351 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:20:32.535418 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 23:20:32.535574 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":1,"thread_start_us":1148,"threads_started":1}
I20260926 23:20:32.537487 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.515405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.16.62.67:40889)
0926 23:20:32.527820 (+ 12415us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.527828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.527860 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:20:32.527959 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.527969 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.528249 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.529041 (+ 792us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:32.529341 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.531491 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.533540 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.534870 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.536664 (+ 1794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.536915 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:32.536986 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 23:20:32.537138 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12251,"thread_start_us":12096,"threads_started":1}
I20260926 23:20:32.538859 323 tablet_service.cc:1467] Processing CreateTablet for tablet f81e62dda14046d5b2efcf23fdc7d1d3 (DEFAULT_TABLE table=test-table [id=268c646c111748138bc8082ba410d597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:20:32.540043 323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f81e62dda14046d5b2efcf23fdc7d1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:20:32.541083 589 tablet_service.cc:1467] Processing CreateTablet for tablet f81e62dda14046d5b2efcf23fdc7d1d3 (DEFAULT_TABLE table=test-table [id=268c646c111748138bc8082ba410d597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:20:32.542459 589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f81e62dda14046d5b2efcf23fdc7d1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:20:32.552083 671 tablet_bootstrap.cc:492] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: Bootstrap starting.
I20260926 23:20:32.553984 672 tablet_bootstrap.cc:492] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: Bootstrap starting.
I20260926 23:20:32.555645 671 tablet_bootstrap.cc:654] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: Neither blocks nor log segments found. Creating new log.
I20260926 23:20:32.556721 671 log.cc:824] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: Log is configured to *not* fsync() on all Append() calls
I20260926 23:20:32.557467 672 tablet_bootstrap.cc:654] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: Neither blocks nor log segments found. Creating new log.
I20260926 23:20:32.557713 673 tablet_bootstrap.cc:492] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: Bootstrap starting.
I20260926 23:20:32.558935 672 log.cc:824] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: Log is configured to *not* fsync() on all Append() calls
I20260926 23:20:32.559089 388 heartbeater.cc:503] Master 127.16.62.126:39965 was elected leader, sending a full tablet report...
I20260926 23:20:32.561010 671 tablet_bootstrap.cc:492] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: No bootstrap required, opened a new log
I20260926 23:20:32.561349 671 ts_tablet_manager.cc:1397] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 23:20:32.561980 673 tablet_bootstrap.cc:654] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: Neither blocks nor log segments found. Creating new log.
I20260926 23:20:32.563227 673 log.cc:824] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: Log is configured to *not* fsync() on all Append() calls
I20260926 23:20:32.566154 672 tablet_bootstrap.cc:492] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: No bootstrap required, opened a new log
I20260926 23:20:32.566368 673 tablet_bootstrap.cc:492] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: No bootstrap required, opened a new log
I20260926 23:20:32.566445 672 ts_tablet_manager.cc:1397] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260926 23:20:32.566612 673 ts_tablet_manager.cc:1397] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 23:20:32.572778 671 raft_consensus.cc:348] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:32.573264 671 raft_consensus.cc:374] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:20:32.573366 671 raft_consensus.cc:729] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcc77497d6b49ef98c2242add6b8170, State: Initialized, Role: FOLLOWER
I20260926 23:20:32.574018 671 consensus_queue.cc:260] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:32.575218 523 heartbeater.cc:503] Master 127.16.62.126:39965 was elected leader, sending a full tablet report...
I20260926 23:20:32.575156 672 raft_consensus.cc:348] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:32.575587 672 raft_consensus.cc:374] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:20:32.575709 672 raft_consensus.cc:729] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f8a7f5c5614bd5b3bc925cc52411a0, State: Initialized, Role: FOLLOWER
I20260926 23:20:32.575958 671 ts_tablet_manager.cc:1428] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 23:20:32.576303 672 consensus_queue.cc:260] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:32.577131 673 raft_consensus.cc:348] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:32.577486 673 raft_consensus.cc:374] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:20:32.577596 673 raft_consensus.cc:729] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787e53d925364e7581f8068528df71bd, State: Initialized, Role: FOLLOWER
I20260926 23:20:32.578069 672 ts_tablet_manager.cc:1428] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 23:20:32.578114 673 consensus_queue.cc:260] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:32.579231 655 heartbeater.cc:503] Master 127.16.62.126:39965 was elected leader, sending a full tablet report...
I20260926 23:20:32.579821 673 ts_tablet_manager.cc:1428] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260926 23:20:32.683296 656 tablet.cc:2369] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:20:32.714275 524 tablet.cc:2369] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:20:32.775347 389 tablet.cc:2369] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:20:32.856383 677 raft_consensus.cc:482] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:20:32.856693 677 raft_consensus.cc:504] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:32.858433 677 leader_election.cc:288] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 787e53d925364e7581f8068528df71bd (127.16.62.67:40889), e0f8a7f5c5614bd5b3bc925cc52411a0 (127.16.62.65:35069)
I20260926 23:20:32.865015 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.858777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:37717 (local address 127.16.62.67:40889)
0926 23:20:32.858996 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.859002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.859866 (+ 864us) server_negotiation.cc:407] Connection header received
0926 23:20:32.860186 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.860192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.860267 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.860388 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:32.860924 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.861615 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.863357 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.864079 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.864281 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.864353 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:32.864672 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 23:20:32.864765 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:20:32.865134 680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.858804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:40889 (local address 127.16.62.66:37717)
0926 23:20:32.859755 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:32.859783 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:32.859944 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:32.860403 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:32.860407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:32.860432 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:32.860786 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.860795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.861758 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.861765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.863212 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.863223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.864298 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.864304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.864497 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.864539 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:20:32.864641 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 23:20:32.864757 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":495,"threads_started":1}
I20260926 23:20:32.865248 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.859156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:50021 (local address 127.16.62.65:35069)
0926 23:20:32.859381 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.859387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.859999 (+ 612us) server_negotiation.cc:407] Connection header received
0926 23:20:32.860242 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.860247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.860336 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.860435 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:20:32.861088 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.861822 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.863511 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.864345 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.864504 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.864555 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:32.864863 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:20:32.864982 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 23:20:32.865303 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.858953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:35069 (local address 127.16.62.66:50021)
0926 23:20:32.859891 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:32.859905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:20:32.860013 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:20:32.860466 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:20:32.860470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:20:32.860499 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:20:32.860927 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.860935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.861956 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.861963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.863349 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:20:32.863360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.864541 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.864553 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:20:32.864718 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.864745 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 23:20:32.864823 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 23:20:32.864939 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":1,"thread_start_us":602,"threads_started":1}
I20260926 23:20:32.866247 609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81e62dda14046d5b2efcf23fdc7d1d3" candidate_uuid: "5bcc77497d6b49ef98c2242add6b8170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787e53d925364e7581f8068528df71bd" is_pre_election: true
I20260926 23:20:32.866426 343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81e62dda14046d5b2efcf23fdc7d1d3" candidate_uuid: "5bcc77497d6b49ef98c2242add6b8170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" is_pre_election: true
I20260926 23:20:32.866839 609 raft_consensus.cc:2454] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bcc77497d6b49ef98c2242add6b8170 in term 0.
I20260926 23:20:32.866991 343 raft_consensus.cc:2454] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bcc77497d6b49ef98c2242add6b8170 in term 0.
I20260926 23:20:32.867599 409 leader_election.cc:302] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bcc77497d6b49ef98c2242add6b8170, 787e53d925364e7581f8068528df71bd; no voters:
I20260926 23:20:32.868057 677 raft_consensus.cc:2781] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:20:32.868249 677 raft_consensus.cc:482] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:20:32.868362 677 raft_consensus.cc:3037] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:20:32.870672 677 raft_consensus.cc:504] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:32.871309 677 leader_election.cc:288] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [CANDIDATE]: Term 1 election: Requested vote from peers 787e53d925364e7581f8068528df71bd (127.16.62.67:40889), e0f8a7f5c5614bd5b3bc925cc52411a0 (127.16.62.65:35069)
I20260926 23:20:32.871711 609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81e62dda14046d5b2efcf23fdc7d1d3" candidate_uuid: "5bcc77497d6b49ef98c2242add6b8170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787e53d925364e7581f8068528df71bd"
I20260926 23:20:32.871812 343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81e62dda14046d5b2efcf23fdc7d1d3" candidate_uuid: "5bcc77497d6b49ef98c2242add6b8170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0"
I20260926 23:20:32.871969 609 raft_consensus.cc:3037] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:20:32.872021 343 raft_consensus.cc:3037] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:20:32.875324 343 raft_consensus.cc:2454] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bcc77497d6b49ef98c2242add6b8170 in term 1.
I20260926 23:20:32.875324 609 raft_consensus.cc:2454] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bcc77497d6b49ef98c2242add6b8170 in term 1.
I20260926 23:20:32.875850 410 leader_election.cc:302] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5bcc77497d6b49ef98c2242add6b8170, 787e53d925364e7581f8068528df71bd, e0f8a7f5c5614bd5b3bc925cc52411a0; no voters:
I20260926 23:20:32.876236 677 raft_consensus.cc:2781] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:20:32.876713 677 raft_consensus.cc:686] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 LEADER]: Becoming Leader. State: Replica: 5bcc77497d6b49ef98c2242add6b8170, State: Running, Role: LEADER
I20260926 23:20:32.877190 677 consensus_queue.cc:237] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } }
I20260926 23:20:32.882036 32670 catalog_manager.cc:5495] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bcc77497d6b49ef98c2242add6b8170 (127.16.62.66). New cstate: current_term: 1 leader_uuid: "5bcc77497d6b49ef98c2242add6b8170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bcc77497d6b49ef98c2242add6b8170" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 } health_report { overall_health: UNKNOWN } } }
I20260926 23:20:32.984114 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:32.978939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.16.62.66:34229)
0926 23:20:32.979126 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:32.979131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:32.979148 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:20:32.979297 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:32.979301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:32.979381 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:32.979508 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:20:32.979850 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.980703 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.982136 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:32.982549 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:32.982646 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:32.983063 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:20:32.983547 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:20:32.983652 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:32.983735 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 23:20:32.983801 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:20:32.992262 343 raft_consensus.cc:1261] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 1 FOLLOWER]: Refusing update from remote peer 5bcc77497d6b49ef98c2242add6b8170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:20:32.992268 609 raft_consensus.cc:1261] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [term 1 FOLLOWER]: Refusing update from remote peer 5bcc77497d6b49ef98c2242add6b8170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:20:32.993194 682 consensus_queue.cc:1035] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "787e53d925364e7581f8068528df71bd" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:20:32.993515 677 consensus_queue.cc:1035] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:20:33.004614 685 mvcc.cc:205] Tried to move back new op lower bound from 7333743955926368256 to 7333743955469156352. Current Snapshot: MvccSnapshot[applied={T|T < 7333743955926368256}]
I20260926 23:20:33.033872 32670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:20:33.034160 32670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:20:33.037626 32670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Sending DeleteTablet for 3 replicas of tablet f81e62dda14046d5b2efcf23fdc7d1d3
I20260926 23:20:33.038633 457 tablet_service.cc:1514] Processing DeleteTablet for tablet f81e62dda14046d5b2efcf23fdc7d1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:20:33 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:44888
I20260926 23:20:33.038764 589 tablet_service.cc:1514] Processing DeleteTablet for tablet f81e62dda14046d5b2efcf23fdc7d1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:20:33 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:55048
I20260926 23:20:33.038786 323 tablet_service.cc:1514] Processing DeleteTablet for tablet f81e62dda14046d5b2efcf23fdc7d1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:20:33 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:50430
I20260926 23:20:33.043082 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 32708
I20260926 23:20:33.051296 695 tablet_replica.cc:330] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: stopping tablet replica
I20260926 23:20:33.051669 696 tablet_replica.cc:330] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: stopping tablet replica
I20260926 23:20:33.051865 694 tablet_replica.cc:330] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: stopping tablet replica
I20260926 23:20:33.052284 696 raft_consensus.cc:2229] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:20:33.052479 695 raft_consensus.cc:2229] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:20:33.052796 694 raft_consensus.cc:2229] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:20:33.052863 696 raft_consensus.cc:2258] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:20:33.053512 694 raft_consensus.cc:2258] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:20:33.053596 695 raft_consensus.cc:2258] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:20:33.055025 696 ts_tablet_manager.cc:1905] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:20:33.055423 694 ts_tablet_manager.cc:1905] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:20:33.055678 695 ts_tablet_manager.cc:1905] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:20:33.062647 694 ts_tablet_manager.cc:1918] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:20:33.062635 695 ts_tablet_manager.cc:1918] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:20:33.062799 695 log.cc:1196] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/f81e62dda14046d5b2efcf23fdc7d1d3
I20260926 23:20:33.062881 694 log.cc:1196] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/f81e62dda14046d5b2efcf23fdc7d1d3
I20260926 23:20:33.063465 694 ts_tablet_manager.cc:1939] T f81e62dda14046d5b2efcf23fdc7d1d3 P 5bcc77497d6b49ef98c2242add6b8170: Deleting consensus metadata
I20260926 23:20:33.063465 695 ts_tablet_manager.cc:1939] T f81e62dda14046d5b2efcf23fdc7d1d3 P 787e53d925364e7581f8068528df71bd: Deleting consensus metadata
I20260926 23:20:33.065212 32655 catalog_manager.cc:4841] TS 787e53d925364e7581f8068528df71bd (127.16.62.67:40889): tablet f81e62dda14046d5b2efcf23fdc7d1d3 (table test-table [id=268c646c111748138bc8082ba410d597]) successfully deleted
I20260926 23:20:33.065501 32655 catalog_manager.cc:4841] TS 5bcc77497d6b49ef98c2242add6b8170 (127.16.62.66:34229): tablet f81e62dda14046d5b2efcf23fdc7d1d3 (table test-table [id=268c646c111748138bc8082ba410d597]) successfully deleted
I20260926 23:20:33.067958 696 ts_tablet_manager.cc:1918] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:20:33.068185 696 log.cc:1196] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/f81e62dda14046d5b2efcf23fdc7d1d3
I20260926 23:20:33.068813 696 ts_tablet_manager.cc:1939] T f81e62dda14046d5b2efcf23fdc7d1d3 P e0f8a7f5c5614bd5b3bc925cc52411a0: Deleting consensus metadata
I20260926 23:20:33.070523 32656 catalog_manager.cc:4841] TS e0f8a7f5c5614bd5b3bc925cc52411a0 (127.16.62.65:35069): tablet f81e62dda14046d5b2efcf23fdc7d1d3 (table test-table [id=268c646c111748138bc8082ba410d597]) successfully deleted
I20260926 23:20:33.080169 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:33.044227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.16.62.65:35069)
0926 23:20:33.044399 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:33.044405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:33.061714 (+ 17309us) server_negotiation.cc:407] Connection header received
0926 23:20:33.062114 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:33.062120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:33.062197 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:33.062263 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:33.065270 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:33.066504 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:33.068505 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:33.069194 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:33.069350 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:33.073094 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:33.073108 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:33.073125 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:33.073162 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:33.076806 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:33.077701 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:33.077710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:33.077714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:33.077875 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:33.078287 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:33.078292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:33.078295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:33.078766 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:33.079055 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:33.079375 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 23:20:33.079597 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:20:33.081193 383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:33.173615 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 32708
I20260926 23:20:33.197777 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 392
I20260926 23:20:33.213071 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:33.198565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.16.62.66:34229)
0926 23:20:33.198770 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:33.198776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:33.198797 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:20:33.198891 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:33.198896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:33.198999 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:33.199105 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:33.200383 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:33.201644 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:33.203063 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:33.203752 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:33.203880 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:33.206367 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:33.206384 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:33.206404 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:33.206447 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:33.210245 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:33.210898 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:33.210907 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:33.210911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:33.211094 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:33.211413 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:33.211418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:33.211421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:33.211862 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:33.212229 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:33.212422 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 23:20:33.212638 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:20:33.213827 518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:33.327647 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 392
I20260926 23:20:33.348999 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 526
I20260926 23:20:33.362131 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:33.349738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.16.62.67:40889)
0926 23:20:33.349918 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:20:33.349925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:20:33.349979 (+ 54us) server_negotiation.cc:407] Connection header received
0926 23:20:33.350197 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:20:33.350207 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:20:33.350289 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:20:33.350395 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:20:33.351745 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:33.352545 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:33.353880 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:20:33.354281 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:20:33.354412 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:20:33.356947 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:20:33.356958 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:20:33.356972 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:20:33.357005 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:20:33.359478 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:33.360116 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:33.360123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:33.360126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:33.360306 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:20:33.360669 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:20:33.360675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:20:33.360678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:20:33.361018 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:20:33.361300 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 23:20:33.361594 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 23:20:33.361807 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:20:33.362790 649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:33.449466 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 526
I20260926 23:20:33.466256 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 32638
I20260926 23:20:33.467388 32701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:20:33.584755 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 32638
I20260926 23:20:53.607704 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:39965
--webserver_interface=127.16.62.126
--webserver_port=37721
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:39965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:20:53.935581 702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:53.935920 702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:53.936089 702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:53.943985 702 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 23:20:53.944130 702 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 23:20:53.944242 702 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 23:20:53.944335 702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:20:53.944427 702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:53.944523 702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:53.944604 702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:20:53.944684 702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:20:53.954766 702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:39965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:39965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=37721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:53.956018 702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:53.957330 702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:53.969096 710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:53.969190 708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:53.969136 707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:53.969583 702 server_base.cc:1034] running on GCE node
I20260926 23:20:53.970501 702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:53.971732 702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:53.972941 702 hybrid_clock.cc:648] HybridClock initialized: now 1790464853972888 us; error 27 us; skew 500 ppm
I20260926 23:20:53.973268 702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:53 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464853, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:53.976328 702 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:20:53.977686 702 webserver.cc:460] Webserver started at http://127.16.62.126:37721/ using document root <none> and password file <none>
I20260926 23:20:53.978302 702 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:53.978384 702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:53.983767 702 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:20:53.986492 717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:53.987775 702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:20:53.987944 702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "03c42eba704743a580e11a213a3a26a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "40de4a8c403b3a97b16fc6504f0ed0cb"
server_key_iv: "8d6117d72cc9d5c2181bda4221c5114e"
server_key_version: "encryptionkey@0"
I20260926 23:20:53.988974 702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:54.015514 702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:54.019587 702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:54.019888 702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:54.038249 702 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:39965
I20260926 23:20:54.038283 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:39965 every 8 connection(s)
I20260926 23:20:54.039664 702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:20:54.041345 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 702
I20260926 23:20:54.042400 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:35069
--local_ip_for_outbound_sockets=127.16.62.65
--tserver_master_addrs=127.16.62.126:39965
--webserver_port=36987
--webserver_interface=127.16.62.65
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:20:54.049168 769 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:20:54.054514 769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Bootstrap starting.
I20260926 23:20:54.076002 769 log.cc:824] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Log is configured to *not* fsync() on all Append() calls
I20260926 23:20:54.091427 769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:20:54.092012 769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Bootstrap complete.
I20260926 23:20:54.101325 769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:54.102154 769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c42eba704743a580e11a213a3a26a0, State: Initialized, Role: FOLLOWER
I20260926 23:20:54.102883 769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:54.103125 769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:20:54.103253 769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:20:54.103461 769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 2 FOLLOWER]: Advancing to term 3
I20260926 23:20:54.107045 769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:54.107508 769 leader_election.cc:302] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03c42eba704743a580e11a213a3a26a0; no voters:
I20260926 23:20:54.108183 769 leader_election.cc:288] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 23:20:54.108316 773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 23:20:54.109457 773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [term 3 LEADER]: Becoming Leader. State: Replica: 03c42eba704743a580e11a213a3a26a0, State: Running, Role: LEADER
I20260926 23:20:54.109956 773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } }
I20260926 23:20:54.110258 769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:20:54.112226 775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03c42eba704743a580e11a213a3a26a0. Latest consensus state: current_term: 3 leader_uuid: "03c42eba704743a580e11a213a3a26a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } } }
I20260926 23:20:54.112571 775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:20:54.112403 774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "03c42eba704743a580e11a213a3a26a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c42eba704743a580e11a213a3a26a0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39965 } } }
I20260926 23:20:54.112787 774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:20:54.121373 780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:20:54.131429 780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:20:54.133378 780 catalog_manager.cc:1259] Loaded cluster ID: 450da93f999a45659d7f54aa4f962977
I20260926 23:20:54.133499 780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:20:54.138634 780 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:20:54.140733 780 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Loaded TSK: 0
I20260926 23:20:54.140893 780 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03c42eba704743a580e11a213a3a26a0: Loaded TSK: 1
F20260926 23:20:54.142365 780 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4135b429ad google::LogMessage::Fail() at ??:0
@ 0x7f4135b4588a google::LogMessage::SendToLog() at ??:0
@ 0x7f4135b424de google::LogMessage::Flush() at ??:0
@ 0x7f4135b43a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f41398a67d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f413989573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f414b5711c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f414b570783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f414b5fc2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f414b5fc0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f414b6465e9 std::function<>::operator()() at ??:0
@ 0x7f414b574cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f414b572c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f414b5fa610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f414b5fa439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f413878e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f413879191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4138791799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f413876ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f413ba466db start_thread at ??:0
@ 0x7f41349dc61f clone at ??:0
W20260926 23:20:54.414400 771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:54.414714 771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:54.414881 771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:54.422071 771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:54.422232 771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:54.422408 771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:20:54.432636 771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:35069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=36987
--tserver_master_addrs=127.16.62.126:39965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:54.433887 771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:54.435182 771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:54.448091 818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:54.448055 820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:54.448179 817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:54.453673 771 server_base.cc:1034] running on GCE node
I20260926 23:20:54.454329 771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:54.455266 771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:54.456507 771 hybrid_clock.cc:648] HybridClock initialized: now 1790464854456470 us; error 52 us; skew 500 ppm
I20260926 23:20:54.456812 771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:54 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:54.459920 771 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:20:54.461261 771 webserver.cc:460] Webserver started at http://127.16.62.65:36987/ using document root <none> and password file <none>
I20260926 23:20:54.461867 771 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:54.461963 771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:54.466943 771 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 23:20:54.469394 827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:54.470701 771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:20:54.470825 771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "e0f8a7f5c5614bd5b3bc925cc52411a0"
format_stamp: "Formatted at 2026-09-26 23:20:08 on dist-test-slave-vcv1"
server_key: "0e3b17b3f1591a63c914485a24ad28ec"
server_key_iv: "42eed758b86e764845e8e688f51280e7"
server_key_version: "encryptionkey@0"
I20260926 23:20:54.471690 771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:54.491550 771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:54.495153 771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:54.495368 771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:54.496274 771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:20:54.497442 771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:20:54.497524 771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:54.497601 771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:20:54.497653 771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:54.531723 771 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:35069
I20260926 23:20:54.531824 940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:35069 every 8 connection(s)
I20260926 23:20:54.533239 771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:20:54.538341 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:54.536482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:43261)
0926 23:20:54.537222 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:54.537477 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":273,"threads_started":1}
W20260926 23:20:54.539090 941 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:39965 (0 consecutive failures): Network error: Failed to ping master at 127.16.62.126:39965: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
I20260926 23:20:54.539888 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:54.539563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:59863)
0926 23:20:54.539719 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:54.539790 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 23:20:54.540776 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:54.540451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:37891)
0926 23:20:54.540587 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:54.540676 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:20:54.541060 941 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:20:54.541136 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 771
I20260926 23:20:54.541813 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:34229
--local_ip_for_outbound_sockets=127.16.62.66
--tserver_master_addrs=127.16.62.126:39965
--webserver_port=42315
--webserver_interface=127.16.62.66
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:20:54.864041 944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:54.864319 944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:54.864485 944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:54.871481 944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:54.871577 944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:54.871797 944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:20:54.881240 944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:34229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=42315
--tserver_master_addrs=127.16.62.126:39965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:54.882424 944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:54.883687 944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:54.896282 949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:54.896245 952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:54.896314 950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:54.897230 944 server_base.cc:1034] running on GCE node
I20260926 23:20:54.897823 944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:54.898727 944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:54.899950 944 hybrid_clock.cc:648] HybridClock initialized: now 1790464854899894 us; error 52 us; skew 500 ppm
I20260926 23:20:54.900316 944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:54 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:54.903316 944 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:20:54.904740 944 webserver.cc:460] Webserver started at http://127.16.62.66:42315/ using document root <none> and password file <none>
I20260926 23:20:54.905416 944 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:54.905555 944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:54.910581 944 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 23:20:54.913234 959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:54.914525 944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:20:54.914696 944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "5bcc77497d6b49ef98c2242add6b8170"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "e5500f1db782a9cf8c5652145f415117"
server_key_iv: "33ca35f637fbc692eddf0bf3130d58de"
server_key_version: "encryptionkey@0"
I20260926 23:20:54.915658 944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:54.931389 944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:54.935111 944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:54.935382 944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:54.936338 944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:20:54.937579 944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:20:54.937700 944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:54.937837 944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:20:54.937929 944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:54.970999 944 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:34229
I20260926 23:20:54.971099 1072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:34229 every 8 connection(s)
I20260926 23:20:54.972564 944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:20:54.974705 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 944
I20260926 23:20:54.975473 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:40889
--local_ip_for_outbound_sockets=127.16.62.67
--tserver_master_addrs=127.16.62.126:39965
--webserver_port=33709
--webserver_interface=127.16.62.67
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:20:54.977815 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:54.975806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:47519)
0926 23:20:54.976684 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:54.976930 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":375,"threads_started":1}
W20260926 23:20:54.978564 1073 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:39965 (0 consecutive failures): Network error: Failed to ping master at 127.16.62.126:39965: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
I20260926 23:20:54.979444 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:54.979121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:53167)
0926 23:20:54.979269 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:54.979342 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 23:20:54.980450 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:54.980063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:40835)
0926 23:20:54.980269 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:54.980344 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260926 23:20:54.980764 1073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:20:55.295535 1076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:20:55.295837 1076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:20:55.296083 1076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:20:55.303421 1076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:20:55.303551 1076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:20:55.303759 1076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:20:55.313618 1076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:40889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=33709
--tserver_master_addrs=127.16.62.126:39965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:20:55.314900 1076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:20:55.316319 1076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:20:55.330068 1084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:55.330134 1081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:20:55.330161 1082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:20:55.331024 1076 server_base.cc:1034] running on GCE node
I20260926 23:20:55.331678 1076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:20:55.332685 1076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:20:55.333904 1076 hybrid_clock.cc:648] HybridClock initialized: now 1790464855333851 us; error 54 us; skew 500 ppm
I20260926 23:20:55.334203 1076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:20:55 dist-test-slave-vcv1 krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464855, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:20:55.337307 1076 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:20:55.338742 1076 webserver.cc:460] Webserver started at http://127.16.62.67:33709/ using document root <none> and password file <none>
I20260926 23:20:55.339401 1076 fs_manager.cc:359] Metadata directory not provided
I20260926 23:20:55.339548 1076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:20:55.344789 1076 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:20:55.347524 1091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:55.348842 1076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:20:55.349009 1076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "787e53d925364e7581f8068528df71bd"
format_stamp: "Formatted at 2026-09-26 23:20:09 on dist-test-slave-vcv1"
server_key: "132d6bf632faa60216fb951ebb278014"
server_key_iv: "0dff8567f3e613f8eca4c215bc91138a"
server_key_version: "encryptionkey@0"
I20260926 23:20:55.350033 1076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:20:55.368050 1076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:20:55.371996 1076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:20:55.372377 1076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:20:55.373332 1076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:20:55.374631 1076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:20:55.374774 1076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:55.374899 1076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:20:55.374993 1076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:20:55.408320 1076 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:40889
I20260926 23:20:55.408376 1203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:40889 every 8 connection(s)
I20260926 23:20:55.409718 1076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:20:55.414335 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:55.412606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:52883)
0926 23:20:55.413322 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:55.413565 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":295,"threads_started":1}
W20260926 23:20:55.414919 1204 heartbeater.cc:644] Failed to heartbeat to 127.16.62.126:39965 (0 consecutive failures): Network error: Failed to ping master at 127.16.62.126:39965: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
I20260926 23:20:55.415583 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:55.415292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:51045)
0926 23:20:55.415425 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:55.415490 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 23:20:55.416390 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:55.416053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:60415)
0926 23:20:55.416212 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:55.416285 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 23:20:55.416621 1204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:20:55.418946 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 1076
I20260926 23:20:55.542420 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:55.541638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:33135)
0926 23:20:55.542108 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:55.542193 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":236,"threads_started":1}
I20260926 23:20:55.982116 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:55.981277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:46517)
0926 23:20:55.981788 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:55.981890 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":286,"threads_started":1}
I20260926 23:20:56.417853 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:56.417120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:38565)
0926 23:20:56.417534 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:56.417642 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":191,"threads_started":1}
I20260926 23:20:56.543972 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:56.543255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:46539)
0926 23:20:56.543732 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:56.543811 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":232,"threads_started":1}
I20260926 23:20:56.983695 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:56.983006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:39355)
0926 23:20:56.983438 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:56.983518 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":218,"threads_started":1}
I20260926 23:20:57.419654 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:57.418813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:38967)
0926 23:20:57.419322 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:57.419440 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":224,"threads_started":1}
I20260926 23:20:57.545473 1215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:57.544789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:51351)
0926 23:20:57.545226 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:57.545304 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260926 23:20:57.985327 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:57.984671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:39629)
0926 23:20:57.985087 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:57.985166 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":231,"threads_started":1}
I20260926 23:20:58.421190 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:58.420513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:45597)
0926 23:20:58.420923 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:58.421020 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":219,"threads_started":1}
I20260926 23:20:58.547030 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:58.546342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:58789)
0926 23:20:58.546765 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:58.546843 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":203,"threads_started":1}
I20260926 23:20:58.986927 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:58.986208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:33821)
0926 23:20:58.986652 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:58.986737 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":218,"threads_started":1}
I20260926 23:20:59.422861 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:59.422072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:36615)
0926 23:20:59.422601 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:59.422677 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":254,"threads_started":1}
I20260926 23:20:59.548774 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:59.547998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:52321)
0926 23:20:59.548511 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:59.548595 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":296,"threads_started":1}
I20260926 23:20:59.988588 1222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:20:59.987828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:40265)
0926 23:20:59.988319 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 23:20:59.988400 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":234,"threads_started":1}
I20260926 23:21:00.424476 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:00.423684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:45723)
0926 23:21:00.424230 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:00.424319 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":281,"threads_started":1}
I20260926 23:21:00.550396 1224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:00.549685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:52881)
0926 23:21:00.550151 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:00.550238 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
I20260926 23:21:00.990170 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:00.989478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:50463)
0926 23:21:00.989910 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:00.990009 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":224,"threads_started":1}
I20260926 23:21:01.426108 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:01.425281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:47227)
0926 23:21:01.425840 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:01.425927 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":228,"threads_started":1}
I20260926 23:21:01.552109 1227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:01.551306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:51741)
0926 23:21:01.551821 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:01.551895 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":241,"threads_started":1}
I20260926 23:21:01.991935 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:01.991102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:43565)
0926 23:21:01.991627 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:01.991714 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":227,"threads_started":1}
I20260926 23:21:02.427709 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:02.426968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:42173)
0926 23:21:02.427417 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:02.427491 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":215,"threads_started":1}
I20260926 23:21:02.553787 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:02.553019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:49001)
0926 23:21:02.553553 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:02.553632 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":310,"threads_started":1}
I20260926 23:21:02.993520 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:02.992856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:47771)
0926 23:21:02.993295 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:02.993371 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20260926 23:21:03.429333 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:03.428579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:37335)
0926 23:21:03.429078 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:03.429162 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":247,"threads_started":1}
I20260926 23:21:03.555431 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:03.554732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:51679)
0926 23:21:03.555183 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:03.555265 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":223,"threads_started":1}
I20260926 23:21:03.995079 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:03.994350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:56717)
0926 23:21:03.994829 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:03.994908 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
I20260926 23:21:04.430913 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:04.430195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:50103)
0926 23:21:04.430659 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:04.430750 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260926 23:21:04.557029 1236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:04.556313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:60947)
0926 23:21:04.556769 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:04.556859 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260926 23:21:04.996809 1237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:04.996082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:42499)
0926 23:21:04.996566 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:04.996648 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":278,"threads_started":1}
I20260926 23:21:05.432487 1238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:05.431791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:33427)
0926 23:21:05.432246 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:05.432321 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":226,"threads_started":1}
I20260926 23:21:05.558642 1239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:05.557931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:46925)
0926 23:21:05.558372 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:05.558458 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":222,"threads_started":1}
I20260926 23:21:05.998533 1240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:05.997779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:45155)
0926 23:21:05.998266 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:05.998354 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":261,"threads_started":1}
I20260926 23:21:06.434011 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:06.433338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:47811)
0926 23:21:06.433747 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:06.433834 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":206,"threads_started":1}
I20260926 23:21:06.560307 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:06.559541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:45185)
0926 23:21:06.559981 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:06.560080 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":248,"threads_started":1}
I20260926 23:21:07.000394 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:06.999570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:50383)
0926 23:21:07.000068 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:07.000230 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260926 23:21:07.435730 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:07.434952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:53079)
0926 23:21:07.435474 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:07.435552 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":314,"threads_started":1}
I20260926 23:21:07.562088 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:07.561189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:53007)
0926 23:21:07.561746 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:07.561889 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":294,"threads_started":1}
I20260926 23:21:08.001955 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:08.001258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:49253)
0926 23:21:08.001684 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:08.001774 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":211,"threads_started":1}
I20260926 23:21:08.437393 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:08.436672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:57237)
0926 23:21:08.437124 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:08.437216 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20260926 23:21:08.563751 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:08.562905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:33051)
0926 23:21:08.563364 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:08.563468 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":214,"threads_started":1}
I20260926 23:21:09.003495 1249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:09.002802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:38441)
0926 23:21:09.003230 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:09.003324 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":200,"threads_started":1}
I20260926 23:21:09.438984 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:09.438284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:54997)
0926 23:21:09.438723 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:09.438809 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260926 23:21:09.565554 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:09.564861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.65:34985)
0926 23:21:09.565286 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:09.565388 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":198,"threads_started":1}
I20260926 23:21:10.005338 1252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:10.004649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.66:60841)
0926 23:21:10.005044 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:10.005152 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":204,"threads_started":1}
I20260926 23:21:10.426642 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 771
I20260926 23:21:10.440790 1254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:10.439978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39965 (local address 127.16.62.67:51095)
0926 23:21:10.440526 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:10.440616 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.126:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":332,"threads_started":1}
I20260926 23:21:10.448472 1253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:10.427827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.16.62.65:35069)
0926 23:21:10.428471 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:10.428485 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:10.428521 (+ 36us) server_negotiation.cc:407] Connection header received
0926 23:21:10.430034 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:10.430043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:10.430465 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:10.430750 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:10.432230 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:10.433430 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:10.434687 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:10.435402 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:10.435548 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:10.442489 (+ 6941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:10.442497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:10.442512 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:10.442549 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:10.445323 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:10.446008 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:10.446015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:10.446021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:10.446182 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:10.446548 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:10.446554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:10.446557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:10.447033 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:10.447353 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:10.447751 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 23:21:10.448203 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":337,"threads_started":1}
I20260926 23:21:10.449767 936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:10.533820 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 771
I20260926 23:21:10.551895 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 944
I20260926 23:21:10.569456 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:10.553022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.16.62.66:34229)
0926 23:21:10.553730 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:10.553749 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:10.553794 (+ 45us) server_negotiation.cc:407] Connection header received
0926 23:21:10.555453 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:10.555463 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:10.555901 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:10.556282 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:10.557525 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:10.558789 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:10.560115 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:10.560851 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:10.561004 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:10.563429 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:10.563436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:10.563450 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:10.563491 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:10.566485 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:10.567189 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:10.567195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:10.567200 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:10.567357 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:10.567663 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:10.567667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:10.567669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:10.568133 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:10.568434 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:10.568757 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 23:21:10.569170 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":354,"threads_started":1}
I20260926 23:21:10.570801 1068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:10.652534 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 944
I20260926 23:21:10.669862 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 1076
I20260926 23:21:10.688484 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:10.670869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.16.62.67:40889)
0926 23:21:10.671439 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:10.671454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:10.671493 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:21:10.673232 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:10.673240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:10.673659 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:10.673996 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:10.675297 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:10.676651 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:10.677924 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:10.678532 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:10.678670 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:10.682338 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:10.682346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:10.682360 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:10.682397 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:10.685468 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:10.686058 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:10.686064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:10.686069 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:10.686225 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:10.686590 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:10.686594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:10.686596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:10.687114 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:10.687340 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:10.687699 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 23:21:10.688114 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":279,"threads_started":1}
I20260926 23:21:10.689795 1199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:10.772010 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 1076
2026-09-26T23:21:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64386 ms)
[----------] 30 tests from SecurityITest (295392 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:10 dist-test-slave-vcv1 krb5kdc[1262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:10 dist-test-slave-vcv1 krb5kdc[1262](info): set up 2 sockets
Sep 26 23:21:10 dist-test-slave-vcv1 krb5kdc[1262](info): commencing operation
krb5kdc: starting...
W20260926 23:21:11.873220 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:12 dist-test-slave-vcv1 krb5kdc[1262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:21:12.116712 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41671
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:41671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:12.454783 1275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:12.455063 1275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:12.455245 1275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:12.462400 1275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:12.462493 1275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:12.462600 1275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:12.462708 1275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:12.462773 1275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:12.472235 1275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:41671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:12.473493 1275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:12.474778 1275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:12.486929 1280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:12.486954 1283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:12.486994 1281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:12.487612 1275 server_base.cc:1034] running on GCE node
I20260926 23:21:12.488446 1275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:12.489754 1275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:12.490969 1275 hybrid_clock.cc:648] HybridClock initialized: now 1790464872490922 us; error 30 us; skew 500 ppm
I20260926 23:21:12.491302 1275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:12 dist-test-slave-vcv1 krb5kdc[1262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464872, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:12.494274 1275 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:21:12.495741 1275 webserver.cc:460] Webserver started at http://127.16.62.126:40641/ using document root <none> and password file <none>
I20260926 23:21:12.496459 1275 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:12.496560 1275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:12.496812 1275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:12.499609 1275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "6786c75c6c0e433eaf37a2cceca984fe"
format_stamp: "Formatted at 2026-09-26 23:21:12 on dist-test-slave-vcv1"
server_key: "07b1ea8ac746ff24e9f29c201441ec32"
server_key_iv: "e00e8d24124852f2b4293a8cb940c2e7"
server_key_version: "encryptionkey@0"
I20260926 23:21:12.500375 1275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "6786c75c6c0e433eaf37a2cceca984fe"
format_stamp: "Formatted at 2026-09-26 23:21:12 on dist-test-slave-vcv1"
server_key: "07b1ea8ac746ff24e9f29c201441ec32"
server_key_iv: "e00e8d24124852f2b4293a8cb940c2e7"
server_key_version: "encryptionkey@0"
I20260926 23:21:12.505234 1275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:21:12.508373 1290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:12.509665 1275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:21:12.509843 1275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "6786c75c6c0e433eaf37a2cceca984fe"
format_stamp: "Formatted at 2026-09-26 23:21:12 on dist-test-slave-vcv1"
server_key: "07b1ea8ac746ff24e9f29c201441ec32"
server_key_iv: "e00e8d24124852f2b4293a8cb940c2e7"
server_key_version: "encryptionkey@0"
I20260926 23:21:12.510030 1275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:12.533052 1275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:12.536834 1275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:12.537141 1275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:12.555104 1275 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:41671
I20260926 23:21:12.555141 1342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:41671 every 8 connection(s)
I20260926 23:21:12.556406 1275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:12.558758 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 1275
I20260926 23:21:12.558971 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:12.559805 16633 external_mini_cluster.cc:1428] Setting key 2d9bc0a0ed6cd50ec3d8b60a3e6bc618
I20260926 23:21:12.560648 1343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:12.572654 1343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe: Bootstrap starting.
Sep 26 23:21:12 dist-test-slave-vcv1 krb5kdc[1262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:12.576335 1343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:12.577344 1343 log.cc:824] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:12.580003 1343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe: No bootstrap required, opened a new log
I20260926 23:21:12.581408 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:12.561830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.16.62.126:41671)
0926 23:21:12.562540 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:12.562551 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:12.562598 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:21:12.564191 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:12.564221 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:12.564583 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:12.564972 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:12.566441 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:12.567730 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:12.569079 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:12.569753 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:12.569973 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:12.574353 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:12.574361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:12.574373 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:12.574407 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:12.577457 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:12.578120 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:12.578143 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:12.578152 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:12.578320 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:12.578701 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:12.578706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:12.578709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:12.579162 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:12.579474 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:12.579921 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 23:21:12.580458 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":326,"threads_started":1}
I20260926 23:21:12.587303 1343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6786c75c6c0e433eaf37a2cceca984fe" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41671 } }
I20260926 23:21:12.587668 1343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:12.587770 1343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6786c75c6c0e433eaf37a2cceca984fe, State: Initialized, Role: FOLLOWER
I20260926 23:21:12.588366 1343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6786c75c6c0e433eaf37a2cceca984fe" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41671 } }
I20260926 23:21:12.588608 1343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:12.588742 1343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:12.588915 1343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:12.590929 1343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6786c75c6c0e433eaf37a2cceca984fe" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41671 } }
I20260926 23:21:12.591388 1343 leader_election.cc:302] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6786c75c6c0e433eaf37a2cceca984fe; no voters:
I20260926 23:21:12.592010 1343 leader_election.cc:288] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:12.592128 1348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:12.593010 1348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [term 1 LEADER]: Becoming Leader. State: Replica: 6786c75c6c0e433eaf37a2cceca984fe, State: Running, Role: LEADER
I20260926 23:21:12.593591 1348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6786c75c6c0e433eaf37a2cceca984fe" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41671 } }
I20260926 23:21:12.594256 1343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:12.596326 1350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6786c75c6c0e433eaf37a2cceca984fe. Latest consensus state: current_term: 1 leader_uuid: "6786c75c6c0e433eaf37a2cceca984fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6786c75c6c0e433eaf37a2cceca984fe" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41671 } } }
I20260926 23:21:12.596660 1350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:12.597101 1349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6786c75c6c0e433eaf37a2cceca984fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6786c75c6c0e433eaf37a2cceca984fe" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41671 } } }
I20260926 23:21:12.597676 1349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:12.598941 1354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:12.603909 1354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:12.612282 1354 catalog_manager.cc:1347] Generated new cluster ID: 0759b19a714f4979aa78aa84eeab0b5b
I20260926 23:21:12.612391 1354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:12.624343 1354 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:12.625581 1354 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:12.634438 1354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe: Generated new TSK 0
I20260926 23:21:12.635136 1354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:21:12.699781 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41671
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:13.032737 1369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:13.032992 1369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:13.033157 1369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:13.040092 1369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:13.040237 1369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:13.040391 1369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:21:13.049808 1369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:13.050974 1369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:13.052227 1369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:13.065083 1374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:13.065022 1375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:13.065481 1377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:13.065819 1369 server_base.cc:1034] running on GCE node
I20260926 23:21:13.066390 1369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:13.067288 1369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:13.068518 1369 hybrid_clock.cc:648] HybridClock initialized: now 1790464873068481 us; error 46 us; skew 500 ppm
I20260926 23:21:13.068861 1369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:13 dist-test-slave-vcv1 krb5kdc[1262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464873, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:13.071930 1369 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:21:13.073439 1369 webserver.cc:460] Webserver started at http://127.16.62.65:33983/ using document root <none> and password file <none>
I20260926 23:21:13.074102 1369 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:13.074244 1369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:13.074553 1369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:13.077564 1369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "b2e0f164158545b0ac4e4c8b14721402"
format_stamp: "Formatted at 2026-09-26 23:21:13 on dist-test-slave-vcv1"
server_key: "577ef9fcb61805cd23795868acadd93d"
server_key_iv: "90b24c571873a3341c363e62767d2fdc"
server_key_version: "encryptionkey@0"
I20260926 23:21:13.078325 1369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "b2e0f164158545b0ac4e4c8b14721402"
format_stamp: "Formatted at 2026-09-26 23:21:13 on dist-test-slave-vcv1"
server_key: "577ef9fcb61805cd23795868acadd93d"
server_key_iv: "90b24c571873a3341c363e62767d2fdc"
server_key_version: "encryptionkey@0"
I20260926 23:21:13.083649 1369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:21:13.087121 1384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:13.088459 1369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:21:13.088627 1369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "b2e0f164158545b0ac4e4c8b14721402"
format_stamp: "Formatted at 2026-09-26 23:21:13 on dist-test-slave-vcv1"
server_key: "577ef9fcb61805cd23795868acadd93d"
server_key_iv: "90b24c571873a3341c363e62767d2fdc"
server_key_version: "encryptionkey@0"
I20260926 23:21:13.088809 1369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:13.111836 1369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:13.116329 1369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:13.116555 1369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:13.117412 1369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:13.118623 1369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:13.118695 1369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:13.118774 1369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:13.118818 1369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:13.150108 1369 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:40789
I20260926 23:21:13.150189 1496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:40789 every 8 connection(s)
I20260926 23:21:13.151396 1369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:21:13.156252 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 1369
I20260926 23:21:13.156458 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:21:13.157100 16633 external_mini_cluster.cc:1428] Setting key 7d54d3d69c322fe7095372428687f317
Sep 26 23:21:13 dist-test-slave-vcv1 krb5kdc[1262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464873, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:13.172317 1499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:13.153962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:49919 (local address 127.16.62.126:41671)
0926 23:21:13.154440 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:13.154444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:13.155209 (+ 765us) server_negotiation.cc:407] Connection header received
0926 23:21:13.157205 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:13.157211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:13.157296 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:13.157407 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:13.159055 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:13.159742 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:13.161274 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:13.161679 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:13.161809 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:13.165662 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:13.165667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:13.165671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:13.165701 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:13.169162 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:13.169851 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:13.169859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:13.169865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:13.170015 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:13.170418 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:13.170425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:13.170431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:13.170789 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:13.170938 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:13.171590 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 23:21:13.171824 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":252,"threads_started":1}
I20260926 23:21:13.172947 1500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:13.154393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41671 (local address 127.16.62.65:49919)
0926 23:21:13.155056 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:13.155097 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:13.156938 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:13.157621 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:13.157635 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:13.158201 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:13.158900 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:13.158920 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:13.159893 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:13.159903 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:13.161137 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:13.161150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:13.161890 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:13.161898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:13.162125 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:13.162802 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:21:13.162822 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:13.165467 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:13.169303 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:13.169310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:13.169322 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:13.169729 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:13.170130 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:13.170133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:13.170135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:13.170298 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:13.170901 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:13.170907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:13.171215 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 23:21:13.171427 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 23:21:13.171783 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
I20260926 23:21:13.174481 1497 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41671
I20260926 23:21:13.174803 1497 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:13.175657 1497 heartbeater.cc:511] Master 127.16.62.126:41671 requested a full tablet report, sending...
I20260926 23:21:13.178431 1307 ts_manager.cc:194] Registered new tserver with Master: b2e0f164158545b0ac4e4c8b14721402 (127.16.62.65:40789)
I20260926 23:21:13.180716 1307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:49919
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:21:13.215371 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41671
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:13.535934 1503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:13.536290 1503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:13.536465 1503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:13.543704 1503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:13.543864 1503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:13.544056 1503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:21:13.553843 1503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:13.555449 1503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:13.557145 1503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:13.571861 1509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:13.572197 1508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:13.572258 1511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:13.573858 1503 server_base.cc:1034] running on GCE node
I20260926 23:21:13.574456 1503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:13.575446 1503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:13.576678 1503 hybrid_clock.cc:648] HybridClock initialized: now 1790464873576632 us; error 39 us; skew 500 ppm
I20260926 23:21:13.576989 1503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:13 dist-test-slave-vcv1 krb5kdc[1262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464873, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:13.580022 1503 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:21:13.581537 1503 webserver.cc:460] Webserver started at http://127.16.62.66:35427/ using document root <none> and password file <none>
I20260926 23:21:13.582280 1503 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:13.582448 1503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:13.582768 1503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:13.585772 1503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "0d61d1130cc24f04b922ffd27d5e4ab9"
format_stamp: "Formatted at 2026-09-26 23:21:13 on dist-test-slave-vcv1"
server_key: "5de3027b67b0acb22062825e64e932b9"
server_key_iv: "f4cb9fb4e555d9dcbb22c5811178388c"
server_key_version: "encryptionkey@0"
I20260926 23:21:13.586509 1503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "0d61d1130cc24f04b922ffd27d5e4ab9"
format_stamp: "Formatted at 2026-09-26 23:21:13 on dist-test-slave-vcv1"
server_key: "5de3027b67b0acb22062825e64e932b9"
server_key_iv: "f4cb9fb4e555d9dcbb22c5811178388c"
server_key_version: "encryptionkey@0"
I20260926 23:21:13.591403 1503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:21:13.594935 1518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:13.596324 1503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 23:21:13.596486 1503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "0d61d1130cc24f04b922ffd27d5e4ab9"
format_stamp: "Formatted at 2026-09-26 23:21:13 on dist-test-slave-vcv1"
server_key: "5de3027b67b0acb22062825e64e932b9"
server_key_iv: "f4cb9fb4e555d9dcbb22c5811178388c"
server_key_version: "encryptionkey@0"
I20260926 23:21:13.596685 1503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:13.628784 1503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:13.634032 1503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:13.634372 1503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:13.635618 1503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:13.637440 1503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:13.637542 1503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:13.637671 1503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:13.637742 1503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:13.671022 1503 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:38741
I20260926 23:21:13.671109 1630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:38741 every 8 connection(s)
I20260926 23:21:13.672386 1503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:21:13.677373 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 1503
I20260926 23:21:13.677579 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:21:13.678118 16633 external_mini_cluster.cc:1428] Setting key 77c928514d9a86980a48a8744ec31893
Sep 26 23:21:13 dist-test-slave-vcv1 krb5kdc[1262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464873, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:13.696262 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:13.674993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:34297 (local address 127.16.62.126:41671)
0926 23:21:13.675471 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:13.675475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:13.676358 (+ 883us) server_negotiation.cc:407] Connection header received
0926 23:21:13.678748 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:13.678755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:13.678845 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:13.678958 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:13.680891 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:13.681593 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:13.684101 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:13.684573 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:13.684678 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:13.688823 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:13.688828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:13.688831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:13.688863 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:13.692695 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:13.693669 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:13.693673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:13.693676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:13.693802 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:13.694380 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:13.694384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:13.694386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:13.694652 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:13.694874 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:13.695750 (+ 876us) server_negotiation.cc:299] Negotiation successful
0926 23:21:13.695917 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":261,"threads_started":1}
I20260926 23:21:13.697734 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:13.675433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41671 (local address 127.16.62.66:34297)
0926 23:21:13.676208 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:13.676252 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:13.678434 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:13.679276 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:13.679285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:13.679904 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:13.680712 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:13.680739 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:13.681763 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:13.681771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:13.683950 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:13.683967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:13.684796 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:13.684805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:13.685137 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:13.685867 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:21:13.685900 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:13.688588 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:13.692873 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:13.692885 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:13.692904 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:13.693550 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:13.693990 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:13.693997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:13.694004 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:13.694242 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:13.694874 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:13.694885 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:13.695337 (+ 452us) client_negotiation.cc:769] Sending connection context
0926 23:21:13.695654 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 23:21:13.696259 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":286,"threads_started":1}
I20260926 23:21:13.699328 1631 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41671
I20260926 23:21:13.699661 1631 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:13.700598 1631 heartbeater.cc:511] Master 127.16.62.126:41671 requested a full tablet report, sending...
I20260926 23:21:13.702392 1307 ts_manager.cc:194] Registered new tserver with Master: 0d61d1130cc24f04b922ffd27d5e4ab9 (127.16.62.66:38741)
I20260926 23:21:13.703715 1307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:34297
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:21:13.739686 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41671
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:14.066208 1637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:14.066545 1637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:14.066721 1637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:14.073879 1637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:14.074040 1637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:14.074213 1637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:21:14.084184 1637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:14.085491 1637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:14.086870 1637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:14.099867 1642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:14.099876 1643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:14.100183 1645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:14.100752 1637 server_base.cc:1034] running on GCE node
I20260926 23:21:14.101318 1637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:14.102185 1637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:14.103376 1637 hybrid_clock.cc:648] HybridClock initialized: now 1790464874103321 us; error 48 us; skew 500 ppm
I20260926 23:21:14.103682 1637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:14 dist-test-slave-vcv1 krb5kdc[1262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464874, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:14.106594 1637 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:21:14.107874 1637 webserver.cc:460] Webserver started at http://127.16.62.67:38149/ using document root <none> and password file <none>
I20260926 23:21:14.108567 1637 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:14.108704 1637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:14.109011 1637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:14.111771 1637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "8c28d6fc37da41e0953c6929aa7606fb"
format_stamp: "Formatted at 2026-09-26 23:21:14 on dist-test-slave-vcv1"
server_key: "3d6dc5e5e22ef9a2e083fb0ecdcb4f48"
server_key_iv: "85c19917296eca00dd9b629606ec121e"
server_key_version: "encryptionkey@0"
I20260926 23:21:14.112563 1637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "8c28d6fc37da41e0953c6929aa7606fb"
format_stamp: "Formatted at 2026-09-26 23:21:14 on dist-test-slave-vcv1"
server_key: "3d6dc5e5e22ef9a2e083fb0ecdcb4f48"
server_key_iv: "85c19917296eca00dd9b629606ec121e"
server_key_version: "encryptionkey@0"
I20260926 23:21:14.117403 1637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 23:21:14.120743 1652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:14.121894 1637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:21:14.122061 1637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "8c28d6fc37da41e0953c6929aa7606fb"
format_stamp: "Formatted at 2026-09-26 23:21:14 on dist-test-slave-vcv1"
server_key: "3d6dc5e5e22ef9a2e083fb0ecdcb4f48"
server_key_iv: "85c19917296eca00dd9b629606ec121e"
server_key_version: "encryptionkey@0"
I20260926 23:21:14.122260 1637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:14.145666 1637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:14.149681 1637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:14.149977 1637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:14.151144 1637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:14.152765 1637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:14.152859 1637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:14.152976 1637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:14.153028 1637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:14.184762 1497 heartbeater.cc:503] Master 127.16.62.126:41671 was elected leader, sending a full tablet report...
I20260926 23:21:14.187276 1637 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:41207
I20260926 23:21:14.187353 1764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:41207 every 8 connection(s)
I20260926 23:21:14.188889 1637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:21:14.194789 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 1637
I20260926 23:21:14.195020 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:21:14.195487 16633 external_mini_cluster.cc:1428] Setting key 1747efcfc804d388caa9d124e7e16562
Sep 26 23:21:14 dist-test-slave-vcv1 krb5kdc[1262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464874, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:14.208626 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.191333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:35745 (local address 127.16.62.126:41671)
0926 23:21:14.191556 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.191560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.192757 (+ 1197us) server_negotiation.cc:407] Connection header received
0926 23:21:14.194728 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.194733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.194835 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.194941 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:14.196671 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.197748 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.199509 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.199987 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.200097 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.203419 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:14.203424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:14.203427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:14.203458 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:14.206052 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.206710 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.206714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.206716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.206834 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.207196 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.207199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.207201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.207453 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:14.207565 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.208247 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.208407 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:21:14.209656 1767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.191832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41671 (local address 127.16.62.67:35745)
0926 23:21:14.192604 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:14.192642 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:14.194478 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:14.195179 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:14.195188 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:14.195742 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:14.196480 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.196504 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.197900 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.197907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.199377 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.199389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.200231 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.200239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.200483 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.201141 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:21:14.201163 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:14.203235 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:14.206176 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:14.206182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:14.206194 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:14.206582 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:14.206924 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:14.206927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:14.206929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:14.207088 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:14.207556 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:14.207561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:14.207839 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 23:21:14.208057 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 23:21:14.208492 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":317,"threads_started":1}
I20260926 23:21:14.211140 1765 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41671
I20260926 23:21:14.211424 1765 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:14.212066 1765 heartbeater.cc:511] Master 127.16.62.126:41671 requested a full tablet report, sending...
I20260926 23:21:14.213406 1306 ts_manager.cc:194] Registered new tserver with Master: 8c28d6fc37da41e0953c6929aa7606fb (127.16.62.67:41207)
I20260926 23:21:14.214272 1306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:35745
I20260926 23:21:14.222673 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:21:14.239383 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.227380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.16.62.126:41671)
0926 23:21:14.227584 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.227588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.227980 (+ 392us) server_negotiation.cc:407] Connection header received
0926 23:21:14.228101 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.228105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.228205 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.228283 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:14.229566 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.230271 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.231610 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.231989 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.232083 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.234397 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:14.234402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:14.234405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:14.234433 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:14.236985 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.237698 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.237701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.237703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.237820 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.238213 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.238216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.238218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.238498 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:14.238686 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.238989 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.239147 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:21:14.243247 1306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:21:14.250103 1306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:21:14.283610 1780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.261990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.16.62.67:41207)
0926 23:21:14.274546 (+ 12556us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.274555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.274602 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:21:14.274718 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.274730 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.274997 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.275187 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:14.276520 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.278102 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.281013 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.282403 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.282540 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.282780 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.283078 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.283246 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12377,"thread_start_us":12418,"threads_started":1}
I20260926 23:21:14.283685 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.261754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41207 (local address 127.0.0.1:52402)
0926 23:21:14.273744 (+ 11990us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:14.273779 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:14.273939 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:14.275377 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:14.275389 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:14.275434 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:14.276189 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.276206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.278220 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.278228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.280849 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.280868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.282598 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.282604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.282813 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.282878 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 23:21:14.283022 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 23:21:14.283208 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11752,"mutex_wait_us":37,"thread_start_us":14392,"threads_started":1}
I20260926 23:21:14.283836 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.275186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:38741 (local address 127.0.0.1:43738)
0926 23:21:14.276486 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:14.276508 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:14.276631 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:14.277247 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:14.277252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:14.277280 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:14.277685 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.277696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.279473 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.279478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.281306 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.281319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.282940 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.282946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.283134 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.283170 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:21:14.283249 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 23:21:14.283373 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":77,"thread_start_us":527,"threads_started":1}
I20260926 23:21:14.284448 1781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.274462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.16.62.66:38741)
0926 23:21:14.275224 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.275236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.276610 (+ 1374us) server_negotiation.cc:407] Connection header received
0926 23:21:14.276796 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.276802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.277005 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.277195 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:14.277861 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.279357 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.281463 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.282461 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.282621 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.282809 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.283920 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.284111 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":386,"threads_started":1}
I20260926 23:21:14.284554 1779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.262050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.16.62.65:40789)
0926 23:21:14.272977 (+ 10927us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.272991 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.276167 (+ 3176us) server_negotiation.cc:407] Connection header received
0926 23:21:14.276525 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.276534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.276782 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.276979 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:14.277624 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.279239 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.281697 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.283139 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.283272 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.283465 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.284055 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.284254 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10684,"thread_start_us":10946,"threads_started":1}
I20260926 23:21:14.285509 1782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.261803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40789 (local address 127.0.0.1:38996)
0926 23:21:14.276023 (+ 14220us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:14.276042 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:14.276290 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:14.276975 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:14.276979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:14.277000 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:14.277456 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.277467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.279361 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.279367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.281546 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.281559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.283626 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.283632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.283857 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.283895 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:21:14.284986 (+ 1091us) client_negotiation.cc:240] Negotiation successful
0926 23:21:14.285108 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14046,"mutex_wait_us":74,"spinlock_wait_cycles":17792,"thread_start_us":11559,"threads_started":1}
I20260926 23:21:14.288347 1700 tablet_service.cc:1467] Processing CreateTablet for tablet d48702ec4708484c98ebeb182dae8ee0 (DEFAULT_TABLE table=test-table [id=1914393781e846b6b50ef46de067007d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:14.288367 1566 tablet_service.cc:1467] Processing CreateTablet for tablet d48702ec4708484c98ebeb182dae8ee0 (DEFAULT_TABLE table=test-table [id=1914393781e846b6b50ef46de067007d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:14.288739 1432 tablet_service.cc:1467] Processing CreateTablet for tablet d48702ec4708484c98ebeb182dae8ee0 (DEFAULT_TABLE table=test-table [id=1914393781e846b6b50ef46de067007d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:14.289674 1700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48702ec4708484c98ebeb182dae8ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:14.289743 1566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48702ec4708484c98ebeb182dae8ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:14.289934 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48702ec4708484c98ebeb182dae8ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:14.302578 1783 tablet_bootstrap.cc:492] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: Bootstrap starting.
I20260926 23:21:14.304625 1784 tablet_bootstrap.cc:492] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: Bootstrap starting.
I20260926 23:21:14.306459 1783 tablet_bootstrap.cc:654] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:14.307723 1783 log.cc:824] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:14.308108 1784 tablet_bootstrap.cc:654] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:14.309413 1784 log.cc:824] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:14.309293 1785 tablet_bootstrap.cc:492] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: Bootstrap starting.
I20260926 23:21:14.311457 1783 tablet_bootstrap.cc:492] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: No bootstrap required, opened a new log
I20260926 23:21:14.312173 1783 ts_tablet_manager.cc:1397] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 23:21:14.312361 1784 tablet_bootstrap.cc:492] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: No bootstrap required, opened a new log
I20260926 23:21:14.312665 1784 ts_tablet_manager.cc:1397] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260926 23:21:14.313565 1785 tablet_bootstrap.cc:654] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:14.314721 1785 log.cc:824] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:14.317634 1785 tablet_bootstrap.cc:492] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: No bootstrap required, opened a new log
I20260926 23:21:14.317879 1785 ts_tablet_manager.cc:1397] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 23:21:14.320423 1784 raft_consensus.cc:348] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } }
I20260926 23:21:14.320787 1784 raft_consensus.cc:374] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:14.320886 1784 raft_consensus.cc:729] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d61d1130cc24f04b922ffd27d5e4ab9, State: Initialized, Role: FOLLOWER
I20260926 23:21:14.321494 1784 consensus_queue.cc:260] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } }
I20260926 23:21:14.322582 1631 heartbeater.cc:503] Master 127.16.62.126:41671 was elected leader, sending a full tablet report...
I20260926 23:21:14.323107 1784 ts_tablet_manager.cc:1428] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260926 23:21:14.323951 1783 raft_consensus.cc:348] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } }
I20260926 23:21:14.326869 1783 raft_consensus.cc:374] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:14.326970 1783 raft_consensus.cc:729] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c28d6fc37da41e0953c6929aa7606fb, State: Initialized, Role: FOLLOWER
I20260926 23:21:14.327556 1783 consensus_queue.cc:260] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } }
I20260926 23:21:14.328298 1785 raft_consensus.cc:348] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } }
I20260926 23:21:14.328711 1785 raft_consensus.cc:374] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:14.328827 1785 raft_consensus.cc:729] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e0f164158545b0ac4e4c8b14721402, State: Initialized, Role: FOLLOWER
I20260926 23:21:14.329036 1765 heartbeater.cc:503] Master 127.16.62.126:41671 was elected leader, sending a full tablet report...
I20260926 23:21:14.329375 1783 ts_tablet_manager.cc:1428] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: Time spent starting tablet: real 0.017s user 0.012s sys 0.000s
I20260926 23:21:14.329421 1785 consensus_queue.cc:260] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } }
I20260926 23:21:14.332041 1785 ts_tablet_manager.cc:1428] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 23:21:14.372745 1791 raft_consensus.cc:482] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:21:14.373034 1791 raft_consensus.cc:504] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } }
I20260926 23:21:14.374696 1791 leader_election.cc:288] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c28d6fc37da41e0953c6929aa7606fb (127.16.62.67:41207), 0d61d1130cc24f04b922ffd27d5e4ab9 (127.16.62.66:38741)
I20260926 23:21:14.381678 1781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.375267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:51069 (local address 127.16.62.66:38741)
0926 23:21:14.375499 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.375508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.376285 (+ 777us) server_negotiation.cc:407] Connection header received
0926 23:21:14.376532 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.376536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.376594 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.376707 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:14.377272 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.378027 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.379830 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.380712 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.380846 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.380936 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.381251 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.381386 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:21:14.381678 1780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.375123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:47679 (local address 127.16.62.67:41207)
0926 23:21:14.375381 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.375390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.376249 (+ 859us) server_negotiation.cc:407] Connection header received
0926 23:21:14.376484 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.376492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.376567 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.376634 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:14.377399 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.378171 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.379940 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.380745 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.380927 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.380997 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.381310 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.381423 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 23:21:14.381810 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.375110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:38741 (local address 127.16.62.65:51069)
0926 23:21:14.376122 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:14.376184 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:14.376368 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:14.376738 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:14.376742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:14.376768 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:14.377100 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.377108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.378162 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.378167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.379679 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.379691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.380915 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.380921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.381100 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.381123 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:21:14.381196 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 23:21:14.381350 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":6,"spinlock_wait_cycles":19840,"thread_start_us":584,"threads_started":1}
I20260926 23:21:14.382050 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.374923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41207 (local address 127.16.62.65:47679)
0926 23:21:14.376093 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:14.376112 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:14.376311 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:14.376694 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:14.376698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:14.376722 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:14.377254 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.377266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.378303 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.378307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.379792 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:14.379807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.381013 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.381018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:14.381138 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.381159 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 23:21:14.381270 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 23:21:14.381418 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":47,"thread_start_us":891,"threads_started":1}
I20260926 23:21:14.382814 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48702ec4708484c98ebeb182dae8ee0" candidate_uuid: "b2e0f164158545b0ac4e4c8b14721402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" is_pre_election: true
I20260926 23:21:14.382917 1720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48702ec4708484c98ebeb182dae8ee0" candidate_uuid: "b2e0f164158545b0ac4e4c8b14721402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c28d6fc37da41e0953c6929aa7606fb" is_pre_election: true
I20260926 23:21:14.383333 1586 raft_consensus.cc:2454] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e0f164158545b0ac4e4c8b14721402 in term 0.
I20260926 23:21:14.383406 1720 raft_consensus.cc:2454] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e0f164158545b0ac4e4c8b14721402 in term 0.
I20260926 23:21:14.384078 1388 leader_election.cc:302] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d61d1130cc24f04b922ffd27d5e4ab9, 8c28d6fc37da41e0953c6929aa7606fb, b2e0f164158545b0ac4e4c8b14721402; no voters:
I20260926 23:21:14.384543 1791 raft_consensus.cc:2781] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:21:14.384694 1791 raft_consensus.cc:482] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:21:14.384828 1791 raft_consensus.cc:3037] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:14.387079 1791 raft_consensus.cc:504] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } }
I20260926 23:21:14.387656 1791 leader_election.cc:288] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [CANDIDATE]: Term 1 election: Requested vote from peers 8c28d6fc37da41e0953c6929aa7606fb (127.16.62.67:41207), 0d61d1130cc24f04b922ffd27d5e4ab9 (127.16.62.66:38741)
I20260926 23:21:14.388168 1720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48702ec4708484c98ebeb182dae8ee0" candidate_uuid: "b2e0f164158545b0ac4e4c8b14721402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c28d6fc37da41e0953c6929aa7606fb"
I20260926 23:21:14.388168 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48702ec4708484c98ebeb182dae8ee0" candidate_uuid: "b2e0f164158545b0ac4e4c8b14721402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9"
I20260926 23:21:14.388368 1720 raft_consensus.cc:3037] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:14.388368 1586 raft_consensus.cc:3037] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:14.391542 1586 raft_consensus.cc:2454] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e0f164158545b0ac4e4c8b14721402 in term 1.
I20260926 23:21:14.391569 1720 raft_consensus.cc:2454] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e0f164158545b0ac4e4c8b14721402 in term 1.
I20260926 23:21:14.391952 1386 leader_election.cc:302] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c28d6fc37da41e0953c6929aa7606fb, b2e0f164158545b0ac4e4c8b14721402; no voters:
I20260926 23:21:14.392288 1791 raft_consensus.cc:2781] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:14.392733 1791 raft_consensus.cc:686] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 1 LEADER]: Becoming Leader. State: Replica: b2e0f164158545b0ac4e4c8b14721402, State: Running, Role: LEADER
I20260926 23:21:14.393236 1791 consensus_queue.cc:237] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } }
I20260926 23:21:14.397578 1306 catalog_manager.cc:5495] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2e0f164158545b0ac4e4c8b14721402 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "b2e0f164158545b0ac4e4c8b14721402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e0f164158545b0ac4e4c8b14721402" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 } health_report { overall_health: UNKNOWN } } }
W20260926 23:21:14.403698 1498 tablet.cc:2369] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:21:14.424443 1632 tablet.cc:2369] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:21:14.426640 1779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.421235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.16.62.65:40789)
0926 23:21:14.421440 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.421444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.421464 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:21:14.421578 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.421582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.421662 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.421801 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:21:14.422122 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.422869 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.424273 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.424819 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.424976 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.425594 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:21:14.426141 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:21:14.426257 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.426342 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.426409 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:21:14.434636 1720 raft_consensus.cc:1261] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [term 1 FOLLOWER]: Refusing update from remote peer b2e0f164158545b0ac4e4c8b14721402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:14.434649 1586 raft_consensus.cc:1261] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [term 1 FOLLOWER]: Refusing update from remote peer b2e0f164158545b0ac4e4c8b14721402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:14.435729 1794 consensus_queue.cc:1035] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d61d1130cc24f04b922ffd27d5e4ab9" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 38741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:14.436157 1791 consensus_queue.cc:1035] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c28d6fc37da41e0953c6929aa7606fb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:21:14.440407 1766 tablet.cc:2369] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:21:14.447037 1797 mvcc.cc:205] Tried to move back new op lower bound from 7333744125674323968 to 7333744125518331904. Current Snapshot: MvccSnapshot[applied={T|T < 7333744125674323968}]
I20260926 23:21:14.448771 1799 mvcc.cc:205] Tried to move back new op lower bound from 7333744125674323968 to 7333744125518331904. Current Snapshot: MvccSnapshot[applied={T|T < 7333744125674323968}]
I20260926 23:21:14.459838 1306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:14.460068 1306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:14.463608 1306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6786c75c6c0e433eaf37a2cceca984fe: Sending DeleteTablet for 3 replicas of tablet d48702ec4708484c98ebeb182dae8ee0
I20260926 23:21:14.464684 1700 tablet_service.cc:1514] Processing DeleteTablet for tablet d48702ec4708484c98ebeb182dae8ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:14 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:52402
I20260926 23:21:14.464811 1432 tablet_service.cc:1514] Processing DeleteTablet for tablet d48702ec4708484c98ebeb182dae8ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:14 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:38996
I20260926 23:21:14.465147 1566 tablet_service.cc:1514] Processing DeleteTablet for tablet d48702ec4708484c98ebeb182dae8ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:14 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:43738
I20260926 23:21:14.466868 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 1369
I20260926 23:21:14.480697 1808 tablet_replica.cc:330] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: stopping tablet replica
I20260926 23:21:14.481004 1807 tablet_replica.cc:330] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: stopping tablet replica
I20260926 23:21:14.481204 1808 raft_consensus.cc:2229] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:14.481585 1807 raft_consensus.cc:2229] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:21:14.481771 1808 raft_consensus.cc:2258] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:14.482375 1807 raft_consensus.cc:2258] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:14.483232 1806 tablet_replica.cc:330] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: stopping tablet replica
I20260926 23:21:14.483703 1806 raft_consensus.cc:2229] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:14.484277 1806 raft_consensus.cc:2258] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:14.486299 1808 ts_tablet_manager.cc:1905] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:14.486486 1806 ts_tablet_manager.cc:1905] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:14.484929 1807 ts_tablet_manager.cc:1905] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:14.494273 1806 ts_tablet_manager.cc:1918] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:14.494937 1806 log.cc:1196] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/d48702ec4708484c98ebeb182dae8ee0
I20260926 23:21:14.495601 1806 ts_tablet_manager.cc:1939] T d48702ec4708484c98ebeb182dae8ee0 P 8c28d6fc37da41e0953c6929aa7606fb: Deleting consensus metadata
I20260926 23:21:14.495936 1807 ts_tablet_manager.cc:1918] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 26 23:21:14 dist-test-slave-vcv1 krb5kdc[1262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:21:14.496094 1807 log.cc:1196] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/d48702ec4708484c98ebeb182dae8ee0
I20260926 23:21:14.496672 1807 ts_tablet_manager.cc:1939] T d48702ec4708484c98ebeb182dae8ee0 P b2e0f164158545b0ac4e4c8b14721402: Deleting consensus metadata
I20260926 23:21:14.497983 1292 catalog_manager.cc:4841] TS 8c28d6fc37da41e0953c6929aa7606fb (127.16.62.67:41207): tablet d48702ec4708484c98ebeb182dae8ee0 (table test-table [id=1914393781e846b6b50ef46de067007d]) successfully deleted
I20260926 23:21:14.498379 1294 catalog_manager.cc:4841] TS b2e0f164158545b0ac4e4c8b14721402 (127.16.62.65:40789): tablet d48702ec4708484c98ebeb182dae8ee0 (table test-table [id=1914393781e846b6b50ef46de067007d]) successfully deleted
I20260926 23:21:14.498955 1808 ts_tablet_manager.cc:1918] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:14.499118 1808 log.cc:1196] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/d48702ec4708484c98ebeb182dae8ee0
I20260926 23:21:14.499624 1808 ts_tablet_manager.cc:1939] T d48702ec4708484c98ebeb182dae8ee0 P 0d61d1130cc24f04b922ffd27d5e4ab9: Deleting consensus metadata
I20260926 23:21:14.501053 1294 catalog_manager.cc:4841] TS 0d61d1130cc24f04b922ffd27d5e4ab9 (127.16.62.66:38741): tablet d48702ec4708484c98ebeb182dae8ee0 (table test-table [id=1914393781e846b6b50ef46de067007d]) successfully deleted
I20260926 23:21:14.507818 1779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.468530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.16.62.65:40789)
0926 23:21:14.468758 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.468769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.482522 (+ 13753us) server_negotiation.cc:407] Connection header received
0926 23:21:14.482820 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.482825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.482917 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.482992 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:14.485431 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.486538 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.488687 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.489387 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.489483 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.501587 (+ 12104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:14.501597 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:14.501610 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:14.501658 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:14.504778 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.505781 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.505787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.505789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.505936 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.506373 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.506376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.506378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.506741 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:14.506941 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.507351 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.507531 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 23:21:14.508695 1492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:14.609167 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 1369
I20260926 23:21:14.627585 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 1503
Sep 26 23:21:14 dist-test-slave-vcv1 krb5kdc[1262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:21:14.641487 1781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.628390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.16.62.66:38741)
0926 23:21:14.628551 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.628556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.628573 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:21:14.628736 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.628740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.628801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.628893 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:14.630161 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.630850 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.632089 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.632648 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.632745 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.636662 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:14.636674 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:14.636688 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:14.636720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:14.639250 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.639846 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.639852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.639855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.639971 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.640345 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.640349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.640351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.640701 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:14.640920 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.641096 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.641244 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 23:21:14.642094 1626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:14.732885 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 1503
I20260926 23:21:14.750445 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 1637
Sep 26 23:21:14 dist-test-slave-vcv1 krb5kdc[1262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:21:14.764624 1780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:14.751213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.16.62.67:41207)
0926 23:21:14.751389 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:14.751399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:14.751417 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:21:14.751547 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:14.751551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:14.751613 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:14.751716 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:14.753001 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.753748 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.754923 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:14.755295 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:14.755386 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:14.759713 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:14.759727 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:14.759743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:14.759779 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:14.762330 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.762935 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.762942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.762944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.763081 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:14.763377 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:14.763380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:14.763382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:14.763702 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:14.763903 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:14.764233 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 23:21:14.764392 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:21:14.765254 1760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:14.854169 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 1637
I20260926 23:21:14.870558 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 1275
I20260926 23:21:14.871735 1337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:14.980691 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 1275
2026-09-26T23:21:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4210 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T23:21:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:15Z Disabled control of system clock
I20260926 23:21:15.054333 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:36305
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:36305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:21:15.373199 1817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:15.373481 1817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:15.373628 1817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:15.380812 1817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:15.380927 1817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:15.380996 1817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:15.381106 1817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:15.381184 1817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:15.390775 1817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:36305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:36305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:15.391997 1817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:15.393770 1817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:15.405297 1822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:15.405234 1825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:15.405239 1823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:15.406090 1817 server_base.cc:1034] running on GCE node
I20260926 23:21:15.406795 1817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:15.408010 1817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:15.409241 1817 hybrid_clock.cc:648] HybridClock initialized: now 1790464875409192 us; error 66 us; skew 500 ppm
I20260926 23:21:15.409592 1817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:15.411655 1817 webserver.cc:460] Webserver started at http://127.16.62.126:40223/ using document root <none> and password file <none>
I20260926 23:21:15.412374 1817 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:15.412519 1817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:15.412837 1817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:15.415555 1817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e"
format_stamp: "Formatted at 2026-09-26 23:21:15 on dist-test-slave-vcv1"
server_key: "7f84902600354498f61bdf263eaacd56"
server_key_iv: "303565e9fb23b6c5b94ceaba252693c6"
server_key_version: "encryptionkey@0"
I20260926 23:21:15.416425 1817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e"
format_stamp: "Formatted at 2026-09-26 23:21:15 on dist-test-slave-vcv1"
server_key: "7f84902600354498f61bdf263eaacd56"
server_key_iv: "303565e9fb23b6c5b94ceaba252693c6"
server_key_version: "encryptionkey@0"
I20260926 23:21:15.421186 1817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:21:15.424471 1831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:15.425797 1817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:21:15.425973 1817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e"
format_stamp: "Formatted at 2026-09-26 23:21:15 on dist-test-slave-vcv1"
server_key: "7f84902600354498f61bdf263eaacd56"
server_key_iv: "303565e9fb23b6c5b94ceaba252693c6"
server_key_version: "encryptionkey@0"
I20260926 23:21:15.426185 1817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:15.455389 1817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:15.456246 1817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:15.456537 1817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:15.474514 1817 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:36305
I20260926 23:21:15.474538 1882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:36305 every 8 connection(s)
I20260926 23:21:15.475888 1817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:15.479444 1883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:15.483986 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 1817
I20260926 23:21:15.484289 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:15.484766 16633 external_mini_cluster.cc:1428] Setting key 55aeba0c2a1f6eb2dc31f50c1480e77c
I20260926 23:21:15.490218 1883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e: Bootstrap starting.
I20260926 23:21:15.493124 1883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:15.494066 1883 log.cc:824] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:15.496762 1883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e: No bootstrap required, opened a new log
I20260926 23:21:15.497956 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:15.486633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.16.62.126:36305)
0926 23:21:15.487902 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:15.487912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:15.487952 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:21:15.489677 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:15.489698 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:15.490117 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:15.490496 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:15.491023 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:15.492349 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:15.493779 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:15.494495 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:15.494696 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:15.495827 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:15.495834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:15.495846 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:15.496412 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:15.496489 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:15.496510 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:15.496648 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:15.496843 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 23:21:15.497111 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":626,"threads_started":1}
I20260926 23:21:15.504510 1883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36305 } }
I20260926 23:21:15.504858 1883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:15.504961 1883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6cd3d0aa744d55a3ab92ae7bf8c60e, State: Initialized, Role: FOLLOWER
I20260926 23:21:15.505594 1883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36305 } }
I20260926 23:21:15.505879 1883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:15.506057 1883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:15.506247 1883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:15.508383 1883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36305 } }
I20260926 23:21:15.508858 1883 leader_election.cc:302] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c6cd3d0aa744d55a3ab92ae7bf8c60e; no voters:
I20260926 23:21:15.509457 1883 leader_election.cc:288] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:15.509656 1888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:15.510862 1888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [term 1 LEADER]: Becoming Leader. State: Replica: 7c6cd3d0aa744d55a3ab92ae7bf8c60e, State: Running, Role: LEADER
I20260926 23:21:15.511409 1888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36305 } }
I20260926 23:21:15.511646 1883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:15.514482 1890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c6cd3d0aa744d55a3ab92ae7bf8c60e. Latest consensus state: current_term: 1 leader_uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36305 } } }
I20260926 23:21:15.514995 1890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:15.514772 1889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6cd3d0aa744d55a3ab92ae7bf8c60e" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 36305 } } }
I20260926 23:21:15.515130 1889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:15.517849 1897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:15.522267 1897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:15.530941 1897 catalog_manager.cc:1347] Generated new cluster ID: 89e80c760d46456d85201aff5f5e15f3
I20260926 23:21:15.531068 1897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:15.543478 1897 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:15.545306 1897 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:15.559923 1897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e: Generated new TSK 0
I20260926 23:21:15.561009 1897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:15.565663 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36305
--builtin_ntp_servers=127.16.62.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:21:15.903687 1907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:15.903998 1907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:15.904096 1907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:15.911527 1907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:15.911701 1907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:15.911825 1907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:21:15.921208 1907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:15.922466 1907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:15.924367 1907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:15.937296 1913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:15.937366 1915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:15.937397 1912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:15.938493 1907 server_base.cc:1034] running on GCE node
I20260926 23:21:15.939069 1907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:15.939960 1907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:15.941188 1907 hybrid_clock.cc:648] HybridClock initialized: now 1790464875941152 us; error 52 us; skew 500 ppm
I20260926 23:21:15.941493 1907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:15.943588 1907 webserver.cc:460] Webserver started at http://127.16.62.65:42627/ using document root <none> and password file <none>
I20260926 23:21:15.944339 1907 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:15.944468 1907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:15.944782 1907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:15.947500 1907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7"
format_stamp: "Formatted at 2026-09-26 23:21:15 on dist-test-slave-vcv1"
server_key: "defa93923520b730d09364ea4fcb2df6"
server_key_iv: "2fc0c401fdf94384c09dbabb40d57a0f"
server_key_version: "encryptionkey@0"
I20260926 23:21:15.948357 1907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7"
format_stamp: "Formatted at 2026-09-26 23:21:15 on dist-test-slave-vcv1"
server_key: "defa93923520b730d09364ea4fcb2df6"
server_key_iv: "2fc0c401fdf94384c09dbabb40d57a0f"
server_key_version: "encryptionkey@0"
I20260926 23:21:15.953111 1907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:21:15.956418 1921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:15.957612 1907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:21:15.957789 1907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7"
format_stamp: "Formatted at 2026-09-26 23:21:15 on dist-test-slave-vcv1"
server_key: "defa93923520b730d09364ea4fcb2df6"
server_key_iv: "2fc0c401fdf94384c09dbabb40d57a0f"
server_key_version: "encryptionkey@0"
I20260926 23:21:15.957994 1907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:15.970036 1907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:15.970814 1907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:15.971114 1907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:15.972019 1907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:15.973438 1907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:15.973560 1907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:15.973701 1907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:15.973790 1907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:16.006906 1907 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:40985
I20260926 23:21:16.006995 2033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:40985 every 8 connection(s)
I20260926 23:21:16.008265 1907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:21:16.008950 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 1907
I20260926 23:21:16.009169 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:21:16.009713 16633 external_mini_cluster.cc:1428] Setting key f4d0b9b81f0a9d1afab94ec065e107dc
I20260926 23:21:16.012969 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36305
--builtin_ntp_servers=127.16.62.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:21:16.022673 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.011058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:50999 (local address 127.16.62.126:36305)
0926 23:21:16.011540 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:16.011546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:16.013069 (+ 1523us) server_negotiation.cc:407] Connection header received
0926 23:21:16.014920 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:16.014927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:16.015039 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:16.015197 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:16.016286 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.017501 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.019075 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.019812 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.019963 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.021759 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:16.021763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:16.021767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:16.021784 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:16.021829 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:16.021841 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:16.021917 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:16.022217 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 23:21:16.022357 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":243,"threads_started":1}
I20260926 23:21:16.024101 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.011739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36305 (local address 127.16.62.65:50999)
0926 23:21:16.012873 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:16.012945 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:16.014660 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:16.015379 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:16.015386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:16.015471 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:16.016033 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.016095 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.017672 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.017676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.018923 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.018936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.020079 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.020084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.020373 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.021366 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:21:16.021372 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:16.021427 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:21:16.021551 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:16.021942 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:16.021955 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:16.021966 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:21:16.022135 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 23:21:16.022590 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":371,"threads_started":1}
I20260926 23:21:16.025861 2034 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36305
I20260926 23:21:16.026290 2034 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:16.027117 2034 heartbeater.cc:511] Master 127.16.62.126:36305 requested a full tablet report, sending...
I20260926 23:21:16.029595 1848 ts_manager.cc:194] Registered new tserver with Master: e3999de8f1b04e8d8bba9b0d9013f7b7 (127.16.62.65:40985)
I20260926 23:21:16.031813 1848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.65:50999
W20260926 23:21:16.339370 2038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:16.339622 2038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:16.339707 2038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:16.346784 2038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:16.346875 2038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:16.346979 2038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:21:16.355793 2038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:16.356931 2038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:16.358563 2038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:16.370942 2043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:16.370909 2044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:16.371481 2046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:16.372200 2038 server_base.cc:1034] running on GCE node
I20260926 23:21:16.372748 2038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:16.373637 2038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:16.374862 2038 hybrid_clock.cc:648] HybridClock initialized: now 1790464876374831 us; error 42 us; skew 500 ppm
I20260926 23:21:16.375175 2038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:16.377285 2038 webserver.cc:460] Webserver started at http://127.16.62.66:36281/ using document root <none> and password file <none>
I20260926 23:21:16.377921 2038 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:16.378050 2038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:16.378355 2038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:16.381068 2038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "acbdc5ede1f9442696cf15a7e20e6465"
format_stamp: "Formatted at 2026-09-26 23:21:16 on dist-test-slave-vcv1"
server_key: "fea40eeb95e7717a95f7cd4f1ea801fa"
server_key_iv: "d063dfb5315362542fe9130a6d95fbba"
server_key_version: "encryptionkey@0"
I20260926 23:21:16.381831 2038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "acbdc5ede1f9442696cf15a7e20e6465"
format_stamp: "Formatted at 2026-09-26 23:21:16 on dist-test-slave-vcv1"
server_key: "fea40eeb95e7717a95f7cd4f1ea801fa"
server_key_iv: "d063dfb5315362542fe9130a6d95fbba"
server_key_version: "encryptionkey@0"
I20260926 23:21:16.386541 2038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:21:16.389925 2052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:16.391196 2038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:21:16.391371 2038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "acbdc5ede1f9442696cf15a7e20e6465"
format_stamp: "Formatted at 2026-09-26 23:21:16 on dist-test-slave-vcv1"
server_key: "fea40eeb95e7717a95f7cd4f1ea801fa"
server_key_iv: "d063dfb5315362542fe9130a6d95fbba"
server_key_version: "encryptionkey@0"
I20260926 23:21:16.391577 2038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:16.407163 2038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:16.407950 2038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:16.408303 2038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:16.409201 2038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:16.410588 2038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:16.410705 2038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:16.410842 2038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:16.410929 2038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:16.443465 2038 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:35999
I20260926 23:21:16.443519 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:35999 every 8 connection(s)
I20260926 23:21:16.444772 2038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:21:16.450959 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 2038
I20260926 23:21:16.451138 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:21:16.451711 16633 external_mini_cluster.cc:1428] Setting key d48e24c1bfcd5b50bfdde76534822bd0
I20260926 23:21:16.454658 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36305
--builtin_ntp_servers=127.16.62.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:21:16.458312 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.447198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:54353 (local address 127.16.62.126:36305)
0926 23:21:16.447416 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:16.447422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:16.448898 (+ 1476us) server_negotiation.cc:407] Connection header received
0926 23:21:16.450511 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:16.450514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:16.450578 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:16.450661 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:16.451979 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.452795 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.454928 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.455691 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.455808 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.457596 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:16.457602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:16.457606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:16.457623 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:16.457657 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:16.457669 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:16.457750 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:16.457988 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 23:21:16.458064 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:21:16.459708 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.447753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36305 (local address 127.16.62.66:54353)
0926 23:21:16.448752 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:16.448786 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:16.450291 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:16.450921 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:16.450935 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:16.451048 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:16.451818 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.451844 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.452938 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.452942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.454756 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.454773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.456156 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.456166 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.456504 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.457253 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:21:16.457259 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:16.457313 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:21:16.457412 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:16.457774 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:16.457787 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:16.457796 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:21:16.457973 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 23:21:16.458380 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":327,"threads_started":1}
I20260926 23:21:16.461231 2165 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36305
I20260926 23:21:16.461544 2165 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:16.462440 2165 heartbeater.cc:511] Master 127.16.62.126:36305 requested a full tablet report, sending...
I20260926 23:21:16.463937 1848 ts_manager.cc:194] Registered new tserver with Master: acbdc5ede1f9442696cf15a7e20e6465 (127.16.62.66:35999)
I20260926 23:21:16.464758 1848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.66:54353
W20260926 23:21:16.777449 2168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:16.777752 2168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:16.777905 2168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:16.785009 2168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:16.785128 2168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:16.785292 2168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:21:16.794131 2168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:16.795428 2168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:16.797230 2168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:16.810732 2174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:16.810722 2173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:16.810746 2176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:16.811636 2168 server_base.cc:1034] running on GCE node
I20260926 23:21:16.812358 2168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:16.813447 2168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:16.814697 2168 hybrid_clock.cc:648] HybridClock initialized: now 1790464876814654 us; error 52 us; skew 500 ppm
I20260926 23:21:16.815057 2168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:16.817358 2168 webserver.cc:460] Webserver started at http://127.16.62.67:34387/ using document root <none> and password file <none>
I20260926 23:21:16.818028 2168 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:16.818167 2168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:16.818472 2168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:16.821338 2168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "b3883e86bdb54921aff170cfecdd6f78"
format_stamp: "Formatted at 2026-09-26 23:21:16 on dist-test-slave-vcv1"
server_key: "02edb430ac5eabc146659261b828764f"
server_key_iv: "0d42e6db541ac7dcb560299e4d6bab44"
server_key_version: "encryptionkey@0"
I20260926 23:21:16.822124 2168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "b3883e86bdb54921aff170cfecdd6f78"
format_stamp: "Formatted at 2026-09-26 23:21:16 on dist-test-slave-vcv1"
server_key: "02edb430ac5eabc146659261b828764f"
server_key_iv: "0d42e6db541ac7dcb560299e4d6bab44"
server_key_version: "encryptionkey@0"
I20260926 23:21:16.827061 2168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:21:16.830428 2182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:16.831722 2168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:21:16.831908 2168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "b3883e86bdb54921aff170cfecdd6f78"
format_stamp: "Formatted at 2026-09-26 23:21:16 on dist-test-slave-vcv1"
server_key: "02edb430ac5eabc146659261b828764f"
server_key_iv: "0d42e6db541ac7dcb560299e4d6bab44"
server_key_version: "encryptionkey@0"
I20260926 23:21:16.832127 2168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:16.848915 2168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:16.849659 2168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:16.849951 2168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:16.850837 2168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:16.852269 2168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:16.852382 2168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:16.852504 2168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:16.852602 2168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:16.884640 2168 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:41901
I20260926 23:21:16.884732 2294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:41901 every 8 connection(s)
I20260926 23:21:16.885939 2168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:21:16.890024 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 2168
I20260926 23:21:16.890316 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:21:16.890791 16633 external_mini_cluster.cc:1428] Setting key 28c79e1a867481eb6c4fb84b92025c65
I20260926 23:21:16.899057 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.888528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:38543 (local address 127.16.62.126:36305)
0926 23:21:16.888705 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:16.888711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:16.890354 (+ 1643us) server_negotiation.cc:407] Connection header received
0926 23:21:16.892420 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:16.892424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:16.892523 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:16.892670 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:16.893976 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.894748 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.896634 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.897070 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.897177 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.898392 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:16.898397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:16.898400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:16.898418 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:16.898451 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:16.898462 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:16.898544 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:16.898725 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 23:21:16.898805 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:21:16.900022 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.889046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:36305 (local address 127.16.62.67:38543)
0926 23:21:16.890092 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:16.890135 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:16.892101 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:16.892935 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:16.892945 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:16.893059 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:16.893835 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.893853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.894914 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.894917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.896478 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.896492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.897256 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.897260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.897455 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.898105 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:21:16.898107 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:16.898145 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:21:16.898224 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:16.898544 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:16.898551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:16.898557 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 23:21:16.898694 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 23:21:16.898976 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":356,"threads_started":1}
I20260926 23:21:16.901360 2295 heartbeater.cc:347] Connected to a master server at 127.16.62.126:36305
I20260926 23:21:16.901659 2295 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:16.902354 2295 heartbeater.cc:511] Master 127.16.62.126:36305 requested a full tablet report, sending...
I20260926 23:21:16.903628 1848 ts_manager.cc:194] Registered new tserver with Master: b3883e86bdb54921aff170cfecdd6f78 (127.16.62.67:41901)
I20260926 23:21:16.904537 1848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.67:38543
I20260926 23:21:16.905954 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:21:16.915359 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.909840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.16.62.126:36305)
0926 23:21:16.909994 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:16.909998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:16.910474 (+ 476us) server_negotiation.cc:407] Connection header received
0926 23:21:16.910532 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:16.910536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:16.910610 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:16.910695 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:16.911102 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.911844 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.913287 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.913700 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.913792 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.914718 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:16.914723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:16.914726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:16.914743 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:16.914776 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:16.914787 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:16.914853 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:16.914967 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 23:21:16.915040 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:21:16.919164 1848 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:21:16.925808 1848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:21:16.950610 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.937592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.0.0.1:37454)
0926 23:21:16.938822 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:16.938863 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:16.939019 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:16.940170 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:16.940182 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:16.940226 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:16.940641 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.940658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.943175 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.943184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.946409 (+ 3225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.946431 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.949335 (+ 2904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.949356 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.949646 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.949714 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 23:21:16.949862 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 23:21:16.950111 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"thread_start_us":849,"threads_started":1}
I20260926 23:21:16.952018 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.937781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.16.62.65:40985)
0926 23:21:16.938531 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:16.938541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:16.939119 (+ 578us) server_negotiation.cc:407] Connection header received
0926 23:21:16.939245 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:16.939257 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:16.939542 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:16.939777 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:16.940910 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.942941 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.946698 (+ 3757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.948678 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.948845 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.949169 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:16.951290 (+ 2121us) server_negotiation.cc:299] Negotiation successful
0926 23:21:16.951529 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":300,"threads_started":1}
I20260926 23:21:16.958164 1969 tablet_service.cc:1467] Processing CreateTablet for tablet e98c6e4b1281457caa65ce70fb3c89e6 (DEFAULT_TABLE table=test-table [id=2d03ed59d2674f15b4874d550c3eefc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:16.959493 1969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98c6e4b1281457caa65ce70fb3c89e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:16.960994 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.939224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41901 (local address 127.0.0.1:43504)
0926 23:21:16.953805 (+ 14581us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:16.953826 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:16.953968 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:16.954475 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:16.954482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:16.954513 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:16.954828 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.954837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.956676 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.956683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.958808 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.958821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.960190 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.960198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.960365 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.960408 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:21:16.960517 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 23:21:16.960643 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14422,"mutex_wait_us":26,"thread_start_us":14980,"threads_started":1}
I20260926 23:21:16.961094 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.940251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.16.62.67:41901)
0926 23:21:16.952914 (+ 12663us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:16.952934 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:16.953981 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 23:21:16.954091 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:16.954097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:16.954294 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:16.954475 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:16.955035 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.956555 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.959016 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.959963 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.960123 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.960417 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:16.960554 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 23:21:16.960711 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12453,"thread_start_us":12276,"threads_started":1}
I20260926 23:21:16.962427 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.938990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:35999 (local address 127.0.0.1:32916)
0926 23:21:16.950923 (+ 11933us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:16.950948 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:16.951080 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:16.954827 (+ 3747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:16.954831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:16.954855 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:16.955065 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.955075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.956431 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.956436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.958807 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:16.958821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.961673 (+ 2852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.961679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:16.961861 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.961892 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:21:16.961971 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 23:21:16.962099 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11801,"mutex_wait_us":52,"thread_start_us":14484,"threads_started":1}
I20260926 23:21:16.963562 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:16.939702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.16.62.66:35999)
0926 23:21:16.953413 (+ 13711us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:16.953424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:16.953459 (+ 35us) server_negotiation.cc:407] Connection header received
0926 23:21:16.953565 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:16.953572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:16.953804 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:16.954824 (+ 1020us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:16.955216 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.956297 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.958977 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:16.960665 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:16.960792 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:16.962925 (+ 2133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:16.963001 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 23:21:16.963151 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13517,"thread_start_us":13378,"threads_started":1}
I20260926 23:21:16.964733 2230 tablet_service.cc:1467] Processing CreateTablet for tablet e98c6e4b1281457caa65ce70fb3c89e6 (DEFAULT_TABLE table=test-table [id=2d03ed59d2674f15b4874d550c3eefc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:16.965883 2230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98c6e4b1281457caa65ce70fb3c89e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:16.966738 2100 tablet_service.cc:1467] Processing CreateTablet for tablet e98c6e4b1281457caa65ce70fb3c89e6 (DEFAULT_TABLE table=test-table [id=2d03ed59d2674f15b4874d550c3eefc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:16.968029 2100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98c6e4b1281457caa65ce70fb3c89e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:16.977056 2313 tablet_bootstrap.cc:492] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7: Bootstrap starting.
I20260926 23:21:16.981544 2313 tablet_bootstrap.cc:654] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:16.982028 2315 tablet_bootstrap.cc:492] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: Bootstrap starting.
I20260926 23:21:16.982362 2314 tablet_bootstrap.cc:492] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: Bootstrap starting.
I20260926 23:21:16.984992 2315 tablet_bootstrap.cc:654] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:16.985024 2314 tablet_bootstrap.cc:654] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:16.985965 2314 log.cc:824] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:16.986186 2315 log.cc:824] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:16.986189 2313 log.cc:824] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:16.989216 2315 tablet_bootstrap.cc:492] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: No bootstrap required, opened a new log
I20260926 23:21:16.989257 2314 tablet_bootstrap.cc:492] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: No bootstrap required, opened a new log
I20260926 23:21:16.989477 2315 ts_tablet_manager.cc:1397] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:21:16.989503 2314 ts_tablet_manager.cc:1397] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 23:21:16.989868 2313 tablet_bootstrap.cc:492] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7: No bootstrap required, opened a new log
I20260926 23:21:16.990108 2313 ts_tablet_manager.cc:1397] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260926 23:21:16.996860 2315 raft_consensus.cc:348] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:16.997236 2315 raft_consensus.cc:374] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:16.997362 2315 raft_consensus.cc:729] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acbdc5ede1f9442696cf15a7e20e6465, State: Initialized, Role: FOLLOWER
I20260926 23:21:16.997956 2315 consensus_queue.cc:260] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:16.999047 2165 heartbeater.cc:503] Master 127.16.62.126:36305 was elected leader, sending a full tablet report...
I20260926 23:21:16.999588 2315 ts_tablet_manager.cc:1428] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 23:21:16.999863 2314 raft_consensus.cc:348] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:17.000305 2314 raft_consensus.cc:374] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:17.000415 2314 raft_consensus.cc:729] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3883e86bdb54921aff170cfecdd6f78, State: Initialized, Role: FOLLOWER
I20260926 23:21:17.001116 2314 consensus_queue.cc:260] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:17.001659 2313 raft_consensus.cc:348] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:17.002084 2313 raft_consensus.cc:374] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:17.002204 2313 raft_consensus.cc:729] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3999de8f1b04e8d8bba9b0d9013f7b7, State: Initialized, Role: FOLLOWER
I20260926 23:21:17.002266 2295 heartbeater.cc:503] Master 127.16.62.126:36305 was elected leader, sending a full tablet report...
I20260926 23:21:17.002869 2314 ts_tablet_manager.cc:1428] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 23:21:17.002744 2313 consensus_queue.cc:260] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:17.004832 2313 ts_tablet_manager.cc:1428] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260926 23:21:17.005079 2034 heartbeater.cc:503] Master 127.16.62.126:36305 was elected leader, sending a full tablet report...
W20260926 23:21:17.010499 2035 tablet.cc:2369] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:21:17.137573 2296 tablet.cc:2369] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:21:17.196735 2166 tablet.cc:2369] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:21:17.311033 2319 raft_consensus.cc:482] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:21:17.311434 2319 raft_consensus.cc:504] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:17.313220 2319 leader_election.cc:288] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3999de8f1b04e8d8bba9b0d9013f7b7 (127.16.62.65:40985), b3883e86bdb54921aff170cfecdd6f78 (127.16.62.67:41901)
I20260926 23:21:17.320608 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.313677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:44665 (local address 127.16.62.65:40985)
0926 23:21:17.313892 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:17.313899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:17.314512 (+ 613us) server_negotiation.cc:407] Connection header received
0926 23:21:17.314700 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:17.314705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:17.314795 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:17.314896 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:17.315753 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.316748 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.318761 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.319607 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.319723 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.319770 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:17.320199 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 23:21:17.320329 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:21:17.320619 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.314768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:52473 (local address 127.16.62.67:41901)
0926 23:21:17.315030 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:17.315037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:17.315383 (+ 346us) server_negotiation.cc:407] Connection header received
0926 23:21:17.315522 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:17.315530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:17.315608 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:17.315740 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:17.316367 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.317191 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.318979 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.319775 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.319927 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.319994 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:17.320272 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 23:21:17.320387 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:21:17.320731 2323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.314526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41901 (local address 127.16.62.66:52473)
0926 23:21:17.315277 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.315294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:17.315396 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:17.315767 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:17.315771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:17.315797 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:17.316224 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:17.316237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.317331 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.317335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:17.318804 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:17.318818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.319948 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.319952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:17.320079 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.320160 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 23:21:17.320248 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 23:21:17.320360 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":452,"threads_started":1}
I20260926 23:21:17.320667 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.313677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.16.62.66:44665)
0926 23:21:17.314389 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.314410 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:17.314557 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:17.315121 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:17.315140 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:17.315175 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:17.315577 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:17.315591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.316910 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.316916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:17.318567 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:17.318579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.319800 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.319809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:17.319991 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.320033 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:21:17.320120 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 23:21:17.320294 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":1,"thread_start_us":396,"threads_started":1}
I20260926 23:21:17.321794 2250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98c6e4b1281457caa65ce70fb3c89e6" candidate_uuid: "acbdc5ede1f9442696cf15a7e20e6465" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3883e86bdb54921aff170cfecdd6f78" is_pre_election: true
I20260926 23:21:17.322082 2320 raft_consensus.cc:482] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:21:17.322086 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98c6e4b1281457caa65ce70fb3c89e6" candidate_uuid: "acbdc5ede1f9442696cf15a7e20e6465" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" is_pre_election: true
I20260926 23:21:17.322431 2320 raft_consensus.cc:504] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:17.322646 1989 raft_consensus.cc:2454] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acbdc5ede1f9442696cf15a7e20e6465 in term 0.
I20260926 23:21:17.323181 2250 raft_consensus.cc:2454] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acbdc5ede1f9442696cf15a7e20e6465 in term 0.
I20260926 23:21:17.323323 2055 leader_election.cc:302] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acbdc5ede1f9442696cf15a7e20e6465, e3999de8f1b04e8d8bba9b0d9013f7b7; no voters:
I20260926 23:21:17.323978 2319 raft_consensus.cc:2781] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:21:17.324097 2319 raft_consensus.cc:482] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:21:17.324254 2319 raft_consensus.cc:3037] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:17.324283 2320 leader_election.cc:288] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3999de8f1b04e8d8bba9b0d9013f7b7 (127.16.62.65:40985), acbdc5ede1f9442696cf15a7e20e6465 (127.16.62.66:35999)
I20260926 23:21:17.327574 2319 raft_consensus.cc:504] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:17.328271 2319 leader_election.cc:288] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [CANDIDATE]: Term 1 election: Requested vote from peers e3999de8f1b04e8d8bba9b0d9013f7b7 (127.16.62.65:40985), b3883e86bdb54921aff170cfecdd6f78 (127.16.62.67:41901)
I20260926 23:21:17.328644 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98c6e4b1281457caa65ce70fb3c89e6" candidate_uuid: "acbdc5ede1f9442696cf15a7e20e6465" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7"
I20260926 23:21:17.328851 1989 raft_consensus.cc:3037] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:17.329025 2250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98c6e4b1281457caa65ce70fb3c89e6" candidate_uuid: "acbdc5ede1f9442696cf15a7e20e6465" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3883e86bdb54921aff170cfecdd6f78"
I20260926 23:21:17.329248 2250 raft_consensus.cc:3037] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:17.331534 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.324474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.16.62.67:42147)
0926 23:21:17.324632 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.324650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:17.324754 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:17.325481 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:17.325486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:17.325514 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:17.325726 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:17.325736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.327070 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.327075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:17.328770 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:17.328783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.330716 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.330724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:17.330915 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.330952 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:21:17.331037 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 23:21:17.331189 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"spinlock_wait_cycles":13056}
I20260926 23:21:17.332243 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.324964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:42147 (local address 127.16.62.65:40985)
0926 23:21:17.325183 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:17.325188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:17.325205 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:21:17.325260 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:17.325265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:17.325334 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:17.325423 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:17.325869 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.326936 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.328950 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.330387 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.330551 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.330620 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:17.331826 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0926 23:21:17.331951 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 23:21:17.332290 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.324875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:54375 (local address 127.16.62.66:35999)
0926 23:21:17.325051 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:17.325058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:17.325478 (+ 420us) server_negotiation.cc:407] Connection header received
0926 23:21:17.325724 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:17.325729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:17.325834 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:17.325907 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:17.326461 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.327209 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.329893 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.331116 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.331250 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.331313 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:17.331871 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 23:21:17.331998 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:21:17.332290 1989 raft_consensus.cc:2454] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acbdc5ede1f9442696cf15a7e20e6465 in term 1.
I20260926 23:21:17.332845 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.324544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:35999 (local address 127.16.62.67:54375)
0926 23:21:17.325371 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.325403 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:17.325553 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:17.326037 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:17.326039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:17.326056 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:17.326311 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:17.326317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.327362 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.327370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:17.329655 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:17.329670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.331532 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.331538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:17.331703 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.331733 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:21:17.332422 (+ 689us) client_negotiation.cc:240] Negotiation successful
0926 23:21:17.332577 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":970,"threads_started":1}
I20260926 23:21:17.332887 2055 leader_election.cc:302] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acbdc5ede1f9442696cf15a7e20e6465, e3999de8f1b04e8d8bba9b0d9013f7b7; no voters:
I20260926 23:21:17.332835 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98c6e4b1281457caa65ce70fb3c89e6" candidate_uuid: "b3883e86bdb54921aff170cfecdd6f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" is_pre_election: true
I20260926 23:21:17.333207 1988 raft_consensus.cc:2379] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3883e86bdb54921aff170cfecdd6f78 in current term 1: Already voted for candidate acbdc5ede1f9442696cf15a7e20e6465 in this term.
I20260926 23:21:17.333724 2250 raft_consensus.cc:2454] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acbdc5ede1f9442696cf15a7e20e6465 in term 1.
I20260926 23:21:17.333667 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98c6e4b1281457caa65ce70fb3c89e6" candidate_uuid: "b3883e86bdb54921aff170cfecdd6f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acbdc5ede1f9442696cf15a7e20e6465" is_pre_election: true
I20260926 23:21:17.334110 2120 raft_consensus.cc:2379] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3883e86bdb54921aff170cfecdd6f78 in current term 1: Already voted for candidate acbdc5ede1f9442696cf15a7e20e6465 in this term.
I20260926 23:21:17.334445 2319 raft_consensus.cc:2781] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:17.334914 2319 raft_consensus.cc:686] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 1 LEADER]: Becoming Leader. State: Replica: acbdc5ede1f9442696cf15a7e20e6465, State: Running, Role: LEADER
I20260926 23:21:17.334843 2184 leader_election.cc:302] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3883e86bdb54921aff170cfecdd6f78; no voters: acbdc5ede1f9442696cf15a7e20e6465, e3999de8f1b04e8d8bba9b0d9013f7b7
I20260926 23:21:17.335299 2320 raft_consensus.cc:2726] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 23:21:17.335467 2319 consensus_queue.cc:237] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } }
I20260926 23:21:17.340312 1847 catalog_manager.cc:5495] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 reported cstate change: term changed from 0 to 1, leader changed from <none> to acbdc5ede1f9442696cf15a7e20e6465 (127.16.62.66). New cstate: current_term: 1 leader_uuid: "acbdc5ede1f9442696cf15a7e20e6465" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acbdc5ede1f9442696cf15a7e20e6465" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 35999 } health_report { overall_health: HEALTHY } } }
I20260926 23:21:17.414549 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.409196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.16.62.66:35999)
0926 23:21:17.409387 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:17.409395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:17.409413 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:21:17.409501 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:17.409505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:17.409611 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:17.409761 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:21:17.410154 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.410885 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.412376 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.412848 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.412976 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.413438 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:21:17.413979 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:21:17.414138 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:17.414198 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 23:21:17.414273 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:21:17.422472 1988 raft_consensus.cc:1261] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 1 FOLLOWER]: Refusing update from remote peer acbdc5ede1f9442696cf15a7e20e6465: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:17.422499 2250 raft_consensus.cc:1261] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 1 FOLLOWER]: Refusing update from remote peer acbdc5ede1f9442696cf15a7e20e6465: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:17.423309 2325 consensus_queue.cc:1035] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3883e86bdb54921aff170cfecdd6f78" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:17.423676 2319 consensus_queue.cc:1035] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3999de8f1b04e8d8bba9b0d9013f7b7" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:17.434504 2328 mvcc.cc:205] Tried to move back new op lower bound from 7333744137913307136 to 7333744137569775616. Current Snapshot: MvccSnapshot[applied={T|T < 7333744137913307136}]
I20260926 23:21:17.451294 1847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:17.451573 1847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:17.455382 1847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c6cd3d0aa744d55a3ab92ae7bf8c60e: Sending DeleteTablet for 3 replicas of tablet e98c6e4b1281457caa65ce70fb3c89e6
I20260926 23:21:17.456873 2230 tablet_service.cc:1514] Processing DeleteTablet for tablet e98c6e4b1281457caa65ce70fb3c89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:17 UTC) from {username='slave'} at 127.0.0.1:43504
I20260926 23:21:17.456889 2100 tablet_service.cc:1514] Processing DeleteTablet for tablet e98c6e4b1281457caa65ce70fb3c89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:17 UTC) from {username='slave'} at 127.0.0.1:32916
I20260926 23:21:17.457755 1969 tablet_service.cc:1514] Processing DeleteTablet for tablet e98c6e4b1281457caa65ce70fb3c89e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:17 UTC) from {username='slave'} at 127.0.0.1:37454
I20260926 23:21:17.463668 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 1907
I20260926 23:21:17.493187 2341 tablet_replica.cc:330] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7: stopping tablet replica
I20260926 23:21:17.493923 2341 raft_consensus.cc:2229] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:17.494750 2341 raft_consensus.cc:2258] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:17.497290 2340 tablet_replica.cc:330] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: stopping tablet replica
I20260926 23:21:17.497539 2341 ts_tablet_manager.cc:1905] T e98c6e4b1281457caa65ce70fb3c89e6 P e3999de8f1b04e8d8bba9b0d9013f7b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:17.497786 2340 raft_consensus.cc:2229] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:21:17.498178 2339 tablet_replica.cc:330] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: stopping tablet replica
I20260926 23:21:17.498456 2340 raft_consensus.cc:2258] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:17.498644 2339 raft_consensus.cc:2229] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:17.499650 2339 raft_consensus.cc:2258] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:17.501617 2339 ts_tablet_manager.cc:1905] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:17.502790 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.465250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.16.62.65:40985)
0926 23:21:17.465435 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:17.465442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:17.492479 (+ 27037us) server_negotiation.cc:407] Connection header received
0926 23:21:17.492697 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:17.492703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:17.492820 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:17.492944 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:17.493635 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.494881 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.498816 (+ 3935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.499450 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.499549 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.501130 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:17.501142 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:17.501159 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:17.501894 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:17.501954 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:17.501979 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:17.502063 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:17.502324 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 23:21:17.502432 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:21:17.504082 2029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:17.512300 2340 ts_tablet_manager.cc:1905] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:17.513978 2339 ts_tablet_manager.cc:1918] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:17.514232 2339 log.cc:1196] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/e98c6e4b1281457caa65ce70fb3c89e6
I20260926 23:21:17.514859 2339 ts_tablet_manager.cc:1939] T e98c6e4b1281457caa65ce70fb3c89e6 P b3883e86bdb54921aff170cfecdd6f78: Deleting consensus metadata
I20260926 23:21:17.516938 1834 catalog_manager.cc:4841] TS b3883e86bdb54921aff170cfecdd6f78 (127.16.62.67:41901): tablet e98c6e4b1281457caa65ce70fb3c89e6 (table test-table [id=2d03ed59d2674f15b4874d550c3eefc7]) successfully deleted
I20260926 23:21:17.519047 2340 ts_tablet_manager.cc:1918] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:17.519248 2340 log.cc:1196] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/e98c6e4b1281457caa65ce70fb3c89e6
I20260926 23:21:17.519780 2340 ts_tablet_manager.cc:1939] T e98c6e4b1281457caa65ce70fb3c89e6 P acbdc5ede1f9442696cf15a7e20e6465: Deleting consensus metadata
I20260926 23:21:17.521306 1833 catalog_manager.cc:4841] TS acbdc5ede1f9442696cf15a7e20e6465 (127.16.62.66:35999): tablet e98c6e4b1281457caa65ce70fb3c89e6 (table test-table [id=2d03ed59d2674f15b4874d550c3eefc7]) successfully deleted
I20260926 23:21:17.620824 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 1907
I20260926 23:21:17.639233 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 2038
W20260926 23:21:17.639464 1834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.62.65:40985 (error 108)
I20260926 23:21:17.641681 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.640865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.0.0.1:37472)
0926 23:21:17.641330 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.641458 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":234,"threads_started":1}
W20260926 23:21:17.642073 1834 catalog_manager.cc:4584] TS e3999de8f1b04e8d8bba9b0d9013f7b7 (127.16.62.65:40985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e98c6e4b1281457caa65ce70fb3c89e6: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40985: connect: Connection refused (error 111)
I20260926 23:21:17.647194 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.640047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.16.62.66:35999)
0926 23:21:17.640254 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:17.640260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:17.640285 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:21:17.640533 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:17.640538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:17.640679 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:17.640804 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:17.641286 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.642327 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.644251 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.644717 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.644810 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.645919 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:17.645927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:17.645939 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:17.646508 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:17.646560 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:17.646578 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:17.646653 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:17.646812 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 23:21:17.646889 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:21:17.647871 2160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:17.695454 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.695009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.0.0.1:37486)
0926 23:21:17.695221 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.695303 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 23:21:17.696501 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.696106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.0.0.1:37502)
0926 23:21:17.696289 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.696364 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 23:21:17.757050 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.756618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.0.0.1:37512)
0926 23:21:17.756823 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.756903 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 23:21:17.758121 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.757750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.0.0.1:37514)
0926 23:21:17.757919 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.757987 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 23:21:17.766574 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 2038
I20260926 23:21:17.786000 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 2168
I20260926 23:21:17.793268 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.786782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.16.62.67:41901)
0926 23:21:17.786980 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:17.786987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:17.787086 (+ 99us) server_negotiation.cc:407] Connection header received
0926 23:21:17.787285 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:17.787291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:17.787385 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:17.787504 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:17.787905 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.788770 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.790091 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:17.790509 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:17.790644 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:17.791762 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:17.791770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:17.791785 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:17.792493 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:17.792556 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:17.792579 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:17.792703 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:17.792839 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 23:21:17.792942 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:21:17.793884 2290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:17.838851 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.838362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.0.0.1:37526)
0926 23:21:17.838610 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.838710 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 23:21:17.839993 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:17.839550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:40985 (local address 127.0.0.1:37532)
0926 23:21:17.839786 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:17.839856 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260926 23:21:17.883222 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 2168
I20260926 23:21:17.899864 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 1817
I20260926 23:21:17.901134 1878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:18.009275 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 1817
2026-09-26T23:21:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3027 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:18 dist-test-slave-vcv1 krb5kdc[2351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:18 dist-test-slave-vcv1 krb5kdc[2351](info): set up 2 sockets
Sep 26 23:21:18 dist-test-slave-vcv1 krb5kdc[2351](info): commencing operation
krb5kdc: starting...
W20260926 23:21:19.109359 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:19 dist-test-slave-vcv1 krb5kdc[2351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:21:19.355235 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:42421
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:42421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:19.672557 2364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:19.672835 2364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:19.672993 2364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:19.680274 2364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:19.680374 2364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:19.680482 2364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:19.680593 2364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:19.680658 2364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:19.690394 2364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:42421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:42421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:19.691622 2364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:19.692946 2364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:19.704453 2369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:19.704424 2372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:19.704491 2370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:19.705919 2364 server_base.cc:1034] running on GCE node
I20260926 23:21:19.706641 2364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:19.707939 2364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:19.709178 2364 hybrid_clock.cc:648] HybridClock initialized: now 1790464879709110 us; error 73 us; skew 500 ppm
I20260926 23:21:19.709551 2364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:19 dist-test-slave-vcv1 krb5kdc[2351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464879, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:19.712764 2364 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:21:19.714745 2364 webserver.cc:460] Webserver started at http://127.16.62.126:38967/ using document root <none> and password file <none>
I20260926 23:21:19.715703 2364 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:19.715885 2364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:19.716236 2364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:19.719007 2364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "162af647614e489a9feb7c23490c60fd"
format_stamp: "Formatted at 2026-09-26 23:21:19 on dist-test-slave-vcv1"
server_key: "fd54af77cb7ada6dea72ec475994be29"
server_key_iv: "dc5f692d6f600946d5435457917f23c6"
server_key_version: "encryptionkey@0"
I20260926 23:21:19.719671 2364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "162af647614e489a9feb7c23490c60fd"
format_stamp: "Formatted at 2026-09-26 23:21:19 on dist-test-slave-vcv1"
server_key: "fd54af77cb7ada6dea72ec475994be29"
server_key_iv: "dc5f692d6f600946d5435457917f23c6"
server_key_version: "encryptionkey@0"
I20260926 23:21:19.724538 2364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:21:19.727715 2379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:19.728960 2364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:19.729146 2364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "162af647614e489a9feb7c23490c60fd"
format_stamp: "Formatted at 2026-09-26 23:21:19 on dist-test-slave-vcv1"
server_key: "fd54af77cb7ada6dea72ec475994be29"
server_key_iv: "dc5f692d6f600946d5435457917f23c6"
server_key_version: "encryptionkey@0"
I20260926 23:21:19.729343 2364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:19.750339 2364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:19.755021 2364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:19.755300 2364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:19.773329 2364 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:42421
I20260926 23:21:19.773352 2430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:42421 every 8 connection(s)
I20260926 23:21:19.774631 2364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:19.778053 2431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:19.779279 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 2364
I20260926 23:21:19.779495 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:19.779924 16633 external_mini_cluster.cc:1428] Setting key d77e855de150f047c058c66d73be9403
I20260926 23:21:19.791810 2431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd: Bootstrap starting.
Sep 26 23:21:19 dist-test-slave-vcv1 krb5kdc[2351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:19.795388 2431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:19.796468 2431 log.cc:824] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:19.799438 2431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd: No bootstrap required, opened a new log
I20260926 23:21:19.802714 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:19.781708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.16.62.126:42421)
0926 23:21:19.782485 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:19.782499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:19.782537 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:21:19.784105 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:19.784204 (+ 99us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:19.784587 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:19.784905 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:19.786619 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:19.787836 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:19.789717 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:19.790361 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:19.790509 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:19.795235 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:19.795242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:19.795256 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:19.795289 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:19.798807 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:19.799491 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:19.799499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:19.799513 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:19.799719 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:19.800101 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:19.800106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:19.800110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:19.800651 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:19.801017 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:19.801380 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 23:21:19.801778 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":390,"threads_started":1}
I20260926 23:21:19.807138 2431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162af647614e489a9feb7c23490c60fd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42421 } }
I20260926 23:21:19.807495 2431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:19.807600 2431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162af647614e489a9feb7c23490c60fd, State: Initialized, Role: FOLLOWER
I20260926 23:21:19.808195 2431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162af647614e489a9feb7c23490c60fd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42421 } }
I20260926 23:21:19.808457 2431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:19.808602 2431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:19.808771 2431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:19.811272 2431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162af647614e489a9feb7c23490c60fd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42421 } }
I20260926 23:21:19.811761 2431 leader_election.cc:302] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 162af647614e489a9feb7c23490c60fd; no voters:
I20260926 23:21:19.812390 2431 leader_election.cc:288] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:19.812615 2436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:19.813766 2436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [term 1 LEADER]: Becoming Leader. State: Replica: 162af647614e489a9feb7c23490c60fd, State: Running, Role: LEADER
I20260926 23:21:19.814321 2436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162af647614e489a9feb7c23490c60fd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42421 } }
I20260926 23:21:19.814519 2431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:19.817386 2438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 162af647614e489a9feb7c23490c60fd. Latest consensus state: current_term: 1 leader_uuid: "162af647614e489a9feb7c23490c60fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162af647614e489a9feb7c23490c60fd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42421 } } }
I20260926 23:21:19.817516 2437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "162af647614e489a9feb7c23490c60fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162af647614e489a9feb7c23490c60fd" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 42421 } } }
I20260926 23:21:19.818035 2438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:19.818039 2437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:19.820077 2445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:19.824015 2445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:19.831641 2445 catalog_manager.cc:1347] Generated new cluster ID: 43f3b15bc0e24a89b0d54aeecda00a12
I20260926 23:21:19.831760 2445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:19.843844 2445 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:19.845618 2445 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:19.868741 2445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd: Generated new TSK 0
I20260926 23:21:19.869740 2445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:21:19.930748 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42421
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:20.252832 2457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:20.253099 2457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:20.253208 2457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:20.260354 2457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:20.260464 2457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:20.260576 2457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:21:20.270160 2457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:20.271333 2457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:20.272536 2457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:20.285118 2462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:20.285172 2463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:20.285298 2465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:20.286305 2457 server_base.cc:1034] running on GCE node
I20260926 23:21:20.286875 2457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:20.287859 2457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:20.289084 2457 hybrid_clock.cc:648] HybridClock initialized: now 1790464880289011 us; error 67 us; skew 500 ppm
I20260926 23:21:20.289404 2457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:20 dist-test-slave-vcv1 krb5kdc[2351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464880, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:20.292446 2457 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:21:20.293879 2457 webserver.cc:460] Webserver started at http://127.16.62.65:42951/ using document root <none> and password file <none>
I20260926 23:21:20.294549 2457 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:20.294694 2457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:20.295004 2457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:20.297859 2457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180"
format_stamp: "Formatted at 2026-09-26 23:21:20 on dist-test-slave-vcv1"
server_key: "da0462e66e7f9c3e40e452ed14ae069e"
server_key_iv: "f4e2832f5308982a6a089280e4947116"
server_key_version: "encryptionkey@0"
I20260926 23:21:20.298697 2457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180"
format_stamp: "Formatted at 2026-09-26 23:21:20 on dist-test-slave-vcv1"
server_key: "da0462e66e7f9c3e40e452ed14ae069e"
server_key_iv: "f4e2832f5308982a6a089280e4947116"
server_key_version: "encryptionkey@0"
I20260926 23:21:20.303517 2457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:21:20.306790 2472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:20.307982 2457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 23:21:20.308194 2457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180"
format_stamp: "Formatted at 2026-09-26 23:21:20 on dist-test-slave-vcv1"
server_key: "da0462e66e7f9c3e40e452ed14ae069e"
server_key_iv: "f4e2832f5308982a6a089280e4947116"
server_key_version: "encryptionkey@0"
I20260926 23:21:20.308419 2457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:20.325626 2457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:20.329505 2457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:20.329792 2457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:20.330713 2457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:20.332015 2457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:20.332116 2457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:20.332335 2457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:20.332406 2457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:20.365891 2457 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:41609
I20260926 23:21:20.365943 2584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:41609 every 8 connection(s)
I20260926 23:21:20.367174 2457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:21:20.373034 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 2457
I20260926 23:21:20.373239 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:21:20.373824 16633 external_mini_cluster.cc:1428] Setting key f02e48cc4455b6146ace78c73e842cb4
Sep 26 23:21:20 dist-test-slave-vcv1 krb5kdc[2351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464880, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:20.387665 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:20.369925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:55057 (local address 127.16.62.126:42421)
0926 23:21:20.370393 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:20.370397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:20.371195 (+ 798us) server_negotiation.cc:407] Connection header received
0926 23:21:20.373416 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:20.373421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:20.373493 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:20.373586 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:20.375473 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:20.376368 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:20.377829 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:20.378224 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:20.378351 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:20.381855 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:20.381860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:20.381864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:20.381905 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:20.384881 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:20.385536 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:20.385544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:20.385549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:20.385666 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:20.386078 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:20.386084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:20.386087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:20.386353 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:20.386500 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:20.387104 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 23:21:20.387331 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":253,"threads_started":1}
I20260926 23:21:20.388576 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:20.370300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:42421 (local address 127.16.62.65:55057)
0926 23:21:20.370990 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:20.371032 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:20.373110 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:20.373863 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:20.373876 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:20.374583 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:20.375294 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:20.375319 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:20.376519 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:20.376523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:20.377690 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:20.377702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:20.378439 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:20.378448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:20.378669 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:20.379329 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:21:20.379349 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:20.381642 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:20.384999 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:20.385007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:20.385019 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:20.385421 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:20.385774 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:20.385778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:20.385782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:20.385970 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:20.386472 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:20.386479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:20.386780 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 23:21:20.387025 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 23:21:20.387482 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":265,"threads_started":1}
I20260926 23:21:20.390055 2585 heartbeater.cc:347] Connected to a master server at 127.16.62.126:42421
I20260926 23:21:20.390355 2585 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:20.391192 2585 heartbeater.cc:511] Master 127.16.62.126:42421 requested a full tablet report, sending...
I20260926 23:21:20.393313 2396 ts_manager.cc:194] Registered new tserver with Master: c29bea331c1c4b89ae7ac4bd4a4b3180 (127.16.62.65:41609)
I20260926 23:21:20.394677 2396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:55057
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:21:20.429774 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42421
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:20.755618 2591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:20.755901 2591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:20.756003 2591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:20.763190 2591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:20.763291 2591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:20.763402 2591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:21:20.772650 2591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:20.773869 2591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:20.775130 2591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:20.788818 2599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:20.788827 2597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:20.789168 2596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:20.789800 2591 server_base.cc:1034] running on GCE node
I20260926 23:21:20.790324 2591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:20.791229 2591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:20.792443 2591 hybrid_clock.cc:648] HybridClock initialized: now 1790464880792388 us; error 55 us; skew 500 ppm
I20260926 23:21:20.792742 2591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:20 dist-test-slave-vcv1 krb5kdc[2351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464880, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:20.800500 2591 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:21:20.801859 2591 webserver.cc:460] Webserver started at http://127.16.62.66:35331/ using document root <none> and password file <none>
I20260926 23:21:20.802520 2591 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:20.802650 2591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:20.802963 2591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:20.805759 2591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "a5a84ff9e6b8483390055e471a59b5c1"
format_stamp: "Formatted at 2026-09-26 23:21:20 on dist-test-slave-vcv1"
server_key: "6a72dfc13557cdf656ee785d420d7311"
server_key_iv: "d0297dcfb752c3969caded162c4ed443"
server_key_version: "encryptionkey@0"
I20260926 23:21:20.806548 2591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "a5a84ff9e6b8483390055e471a59b5c1"
format_stamp: "Formatted at 2026-09-26 23:21:20 on dist-test-slave-vcv1"
server_key: "6a72dfc13557cdf656ee785d420d7311"
server_key_iv: "d0297dcfb752c3969caded162c4ed443"
server_key_version: "encryptionkey@0"
I20260926 23:21:20.811450 2591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 23:21:20.814863 2606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:20.816250 2591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:20.816403 2591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "a5a84ff9e6b8483390055e471a59b5c1"
format_stamp: "Formatted at 2026-09-26 23:21:20 on dist-test-slave-vcv1"
server_key: "6a72dfc13557cdf656ee785d420d7311"
server_key_iv: "d0297dcfb752c3969caded162c4ed443"
server_key_version: "encryptionkey@0"
I20260926 23:21:20.816591 2591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:20.840435 2591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:20.844372 2591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:20.844663 2591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:20.845585 2591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:20.846836 2591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:20.846904 2591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:20.847059 2591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:20.847110 2591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:20.879283 2591 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:43883
I20260926 23:21:20.879326 2718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:43883 every 8 connection(s)
I20260926 23:21:20.880628 2591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:21:20.882002 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 2591
I20260926 23:21:20.882205 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:21:20.882647 16633 external_mini_cluster.cc:1428] Setting key 4058f5eb1f7de7dc7cc452776827593b
Sep 26 23:21:20 dist-test-slave-vcv1 krb5kdc[2351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464880, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:20.903193 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:20.883271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:49111 (local address 127.16.62.126:42421)
0926 23:21:20.883488 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:20.883495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:20.884824 (+ 1329us) server_negotiation.cc:407] Connection header received
0926 23:21:20.887693 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:20.887698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:20.887764 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:20.887851 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:20.889879 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:20.890588 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:20.893057 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:20.893459 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:20.893562 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:20.897406 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:20.897410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:20.897414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:20.897447 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:20.900185 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:20.900913 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:20.900918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:20.900922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:20.901067 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:20.901507 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:20.901512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:20.901515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:20.901768 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:20.901949 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:20.902609 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 23:21:20.902830 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:21:20.904222 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:20.883801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:42421 (local address 127.16.62.66:49111)
0926 23:21:20.884640 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:20.884701 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:20.887398 (+ 2697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:20.888113 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:20.888128 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:20.888906 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:20.889702 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:20.889728 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:20.890750 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:20.890759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:20.892902 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:20.892919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:20.893685 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:20.893694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:20.894019 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:20.894815 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:21:20.894835 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:20.897183 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:20.900318 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:20.900327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:20.900343 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:20.900785 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:20.901199 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:20.901204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:20.901208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:20.901386 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:20.901918 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:20.901926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:20.902245 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 23:21:20.902517 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 23:21:20.903039 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":336,"threads_started":1}
I20260926 23:21:20.905789 2719 heartbeater.cc:347] Connected to a master server at 127.16.62.126:42421
I20260926 23:21:20.906101 2719 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:20.906795 2719 heartbeater.cc:511] Master 127.16.62.126:42421 requested a full tablet report, sending...
I20260926 23:21:20.908360 2396 ts_manager.cc:194] Registered new tserver with Master: a5a84ff9e6b8483390055e471a59b5c1 (127.16.62.66:43883)
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
I20260926 23:21:20.909197 2396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:49111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:21:20.939982 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42421
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:21.255649 2724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:21.255925 2724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:21.256016 2724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:21.263139 2724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:21.263240 2724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:21.263350 2724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:21:21.272666 2724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:21.273802 2724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:21.275013 2724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:21.288313 2730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:21.288372 2729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:21.288337 2732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:21.289101 2724 server_base.cc:1034] running on GCE node
I20260926 23:21:21.289683 2724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:21.290673 2724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:21.291908 2724 hybrid_clock.cc:648] HybridClock initialized: now 1790464881291857 us; error 43 us; skew 500 ppm
I20260926 23:21:21.292270 2724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:21 dist-test-slave-vcv1 krb5kdc[2351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464881, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:21.295250 2724 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:21:21.296707 2724 webserver.cc:460] Webserver started at http://127.16.62.67:34285/ using document root <none> and password file <none>
I20260926 23:21:21.297364 2724 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:21.297503 2724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:21.297821 2724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:21.300812 2724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "49c98bb4f4c44fd49c2135115e1958bb"
format_stamp: "Formatted at 2026-09-26 23:21:21 on dist-test-slave-vcv1"
server_key: "0b3fe3287732234cd1edc29083e436f5"
server_key_iv: "faeda6f82e86db6bb29016ad3ecc3557"
server_key_version: "encryptionkey@0"
I20260926 23:21:21.301548 2724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "49c98bb4f4c44fd49c2135115e1958bb"
format_stamp: "Formatted at 2026-09-26 23:21:21 on dist-test-slave-vcv1"
server_key: "0b3fe3287732234cd1edc29083e436f5"
server_key_iv: "faeda6f82e86db6bb29016ad3ecc3557"
server_key_version: "encryptionkey@0"
I20260926 23:21:21.306320 2724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:21:21.309603 2739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:21.310901 2724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:21:21.311077 2724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "49c98bb4f4c44fd49c2135115e1958bb"
format_stamp: "Formatted at 2026-09-26 23:21:21 on dist-test-slave-vcv1"
server_key: "0b3fe3287732234cd1edc29083e436f5"
server_key_iv: "faeda6f82e86db6bb29016ad3ecc3557"
server_key_version: "encryptionkey@0"
I20260926 23:21:21.311273 2724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:21.338186 2724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:21.342206 2724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:21.342497 2724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:21.343457 2724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:21.344830 2724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:21.344933 2724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:21.345083 2724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:21.345163 2724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:21.378130 2724 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:41859
I20260926 23:21:21.378221 2851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:41859 every 8 connection(s)
I20260926 23:21:21.379331 2724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:21:21.382223 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 2724
I20260926 23:21:21.382400 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:21:21.383015 16633 external_mini_cluster.cc:1428] Setting key 2115c9025d180966fbc7e8baa9ce1cdf
Sep 26 23:21:21 dist-test-slave-vcv1 krb5kdc[2351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464881, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:21.398187 2585 heartbeater.cc:503] Master 127.16.62.126:42421 was elected leader, sending a full tablet report...
I20260926 23:21:21.398739 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.381750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:48201 (local address 127.16.62.126:42421)
0926 23:21:21.381923 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.381927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.383326 (+ 1399us) server_negotiation.cc:407] Connection header received
0926 23:21:21.385778 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.385781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.385841 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.385957 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:21.387287 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.388023 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.389511 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.389897 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.389987 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.393112 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:21.393116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:21.393120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:21.393156 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:21.395484 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.396125 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.396130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.396157 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.396312 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.396791 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.396796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.396801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.397192 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:21.397346 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.398160 (+ 814us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.398379 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:21:21.399688 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.382266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:42421 (local address 127.16.62.67:48201)
0926 23:21:21.383143 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:21.383194 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:21.385446 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:21.386140 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:21.386147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:21.386603 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:21.387141 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.387158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.388185 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.388189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.389392 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.389403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.390063 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.390067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.390252 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.390919 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:21:21.390940 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:21.392952 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:21.395601 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:21.395607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:21.395616 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:21.396006 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:21.396445 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:21.396449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:21.396453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:21.396655 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:21.397316 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:21.397322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:21.397689 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 23:21:21.397977 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 23:21:21.398464 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":352,"threads_started":1}
I20260926 23:21:21.400997 2852 heartbeater.cc:347] Connected to a master server at 127.16.62.126:42421
I20260926 23:21:21.401243 2852 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:21.401816 2852 heartbeater.cc:511] Master 127.16.62.126:42421 requested a full tablet report, sending...
I20260926 23:21:21.402968 2396 ts_manager.cc:194] Registered new tserver with Master: 49c98bb4f4c44fd49c2135115e1958bb (127.16.62.67:41859)
I20260926 23:21:21.403669 2396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:48201
I20260926 23:21:21.410084 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:21:21.427016 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.414065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.16.62.126:42421)
0926 23:21:21.414285 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.414290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.414794 (+ 504us) server_negotiation.cc:407] Connection header received
0926 23:21:21.414937 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.414942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.415007 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.415083 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:21.416632 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.417326 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.418759 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.419155 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.419252 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.421992 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:21.421997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:21.422000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:21.422031 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:21.424659 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.425363 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.425367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.425369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.425497 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.425845 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.425848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.425850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.426125 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:21.426355 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.426642 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.426765 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 23:21:21.430799 2396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:21:21.437604 2396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:21:21.471735 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.449326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.16.62.67:41859)
0926 23:21:21.461586 (+ 12260us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.461594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.461644 (+ 50us) server_negotiation.cc:407] Connection header received
0926 23:21:21.461760 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.461770 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.462081 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.463282 (+ 1201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:21.463651 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.465738 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.468959 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.470495 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.470653 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.470900 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.471208 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.471360 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12086,"thread_start_us":16374,"threads_started":1}
I20260926 23:21:21.471870 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.449189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41859 (local address 127.0.0.1:35390)
0926 23:21:21.460636 (+ 11447us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:21.460692 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:21.460905 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:21.462363 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:21.462375 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:21.462419 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:21.463031 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.463047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.465935 (+ 2888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.465944 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.468792 (+ 2848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.468806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.470701 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.470707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.470971 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.471014 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:21:21.471149 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 23:21:21.471344 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11121,"mutex_wait_us":3,"thread_start_us":10950,"threads_started":1}
I20260926 23:21:21.474393 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.449189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:41609 (local address 127.0.0.1:46014)
0926 23:21:21.465762 (+ 16573us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:21.465787 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:21.465924 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:21.467655 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:21.467659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:21.467683 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:21.468086 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.468097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.469711 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.469717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.471704 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.471718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.473590 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.473597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.473759 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.473791 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:21:21.473887 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:21:21.474023 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16454,"mutex_wait_us":450,"thread_start_us":15912,"threads_started":1}
I20260926 23:21:21.474452 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.449603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.16.62.65:41609)
0926 23:21:21.460177 (+ 10574us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.460188 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.466780 (+ 6592us) server_negotiation.cc:407] Connection header received
0926 23:21:21.466892 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.466900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.467160 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.467352 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:21.468346 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.469549 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.471887 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.473351 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.473480 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.473700 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.473944 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.474108 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10354,"thread_start_us":10159,"threads_started":1}
I20260926 23:21:21.475322 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.462562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:43883 (local address 127.0.0.1:47296)
0926 23:21:21.466336 (+ 3774us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:21.466370 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:21.466535 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:21.468112 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:21.468117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:21.468172 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:21.468554 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.468562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.470411 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.470416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.472541 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.472557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.474570 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.474576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.474735 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.474791 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 23:21:21.474883 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:21:21.475018 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3620,"thread_start_us":3460,"threads_started":1}
I20260926 23:21:21.475443 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.461478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.16.62.66:43883)
0926 23:21:21.467677 (+ 6199us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.467684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.467704 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:21:21.467775 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.467780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.467925 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.468123 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:21.468709 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.470237 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.472731 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.474115 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.474258 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.474436 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.474971 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.475139 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6029,"thread_start_us":5913,"threads_started":1}
I20260926 23:21:21.476363 2787 tablet_service.cc:1467] Processing CreateTablet for tablet 72530cb49613415e8c68be57cbcd2ef3 (DEFAULT_TABLE table=test-table [id=92cdb4026f89410f8310ecd8e82b0a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:21.477924 2787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72530cb49613415e8c68be57cbcd2ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:21.478013 2520 tablet_service.cc:1467] Processing CreateTablet for tablet 72530cb49613415e8c68be57cbcd2ef3 (DEFAULT_TABLE table=test-table [id=92cdb4026f89410f8310ecd8e82b0a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:21.479146 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72530cb49613415e8c68be57cbcd2ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:21.479050 2654 tablet_service.cc:1467] Processing CreateTablet for tablet 72530cb49613415e8c68be57cbcd2ef3 (DEFAULT_TABLE table=test-table [id=92cdb4026f89410f8310ecd8e82b0a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:21.480342 2654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72530cb49613415e8c68be57cbcd2ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:21.489886 2870 tablet_bootstrap.cc:492] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: Bootstrap starting.
I20260926 23:21:21.492743 2871 tablet_bootstrap.cc:492] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: Bootstrap starting.
I20260926 23:21:21.493220 2870 tablet_bootstrap.cc:654] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:21.494201 2872 tablet_bootstrap.cc:492] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: Bootstrap starting.
I20260926 23:21:21.494802 2870 log.cc:824] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:21.496389 2871 tablet_bootstrap.cc:654] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:21.497239 2872 tablet_bootstrap.cc:654] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:21.497525 2871 log.cc:824] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:21.498298 2872 log.cc:824] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:21.498831 2870 tablet_bootstrap.cc:492] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: No bootstrap required, opened a new log
I20260926 23:21:21.499146 2870 ts_tablet_manager.cc:1397] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260926 23:21:21.500828 2871 tablet_bootstrap.cc:492] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: No bootstrap required, opened a new log
I20260926 23:21:21.501077 2871 ts_tablet_manager.cc:1397] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260926 23:21:21.502244 2872 tablet_bootstrap.cc:492] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: No bootstrap required, opened a new log
I20260926 23:21:21.502483 2872 ts_tablet_manager.cc:1397] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:21:21.509009 2872 raft_consensus.cc:348] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } }
I20260926 23:21:21.509378 2872 raft_consensus.cc:374] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:21.509505 2872 raft_consensus.cc:729] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a84ff9e6b8483390055e471a59b5c1, State: Initialized, Role: FOLLOWER
I20260926 23:21:21.509964 2870 raft_consensus.cc:348] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } }
I20260926 23:21:21.510113 2872 consensus_queue.cc:260] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } }
I20260926 23:21:21.510514 2870 raft_consensus.cc:374] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:21.510641 2870 raft_consensus.cc:729] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29bea331c1c4b89ae7ac4bd4a4b3180, State: Initialized, Role: FOLLOWER
I20260926 23:21:21.511217 2719 heartbeater.cc:503] Master 127.16.62.126:42421 was elected leader, sending a full tablet report...
I20260926 23:21:21.511415 2870 consensus_queue.cc:260] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } }
I20260926 23:21:21.511932 2872 ts_tablet_manager.cc:1428] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260926 23:21:21.512378 2871 raft_consensus.cc:348] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } }
I20260926 23:21:21.513017 2871 raft_consensus.cc:374] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:21.513152 2871 raft_consensus.cc:729] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c98bb4f4c44fd49c2135115e1958bb, State: Initialized, Role: FOLLOWER
I20260926 23:21:21.513868 2870 ts_tablet_manager.cc:1428] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260926 23:21:21.513929 2871 consensus_queue.cc:260] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } }
I20260926 23:21:21.514989 2852 heartbeater.cc:503] Master 127.16.62.126:42421 was elected leader, sending a full tablet report...
I20260926 23:21:21.515796 2871 ts_tablet_manager.cc:1428] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260926 23:21:21.588285 2877 raft_consensus.cc:482] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:21:21.588608 2877 raft_consensus.cc:504] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } }
I20260926 23:21:21.590318 2877 leader_election.cc:288] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c98bb4f4c44fd49c2135115e1958bb (127.16.62.67:41859), a5a84ff9e6b8483390055e471a59b5c1 (127.16.62.66:43883)
I20260926 23:21:21.598584 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.591670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:41185 (local address 127.16.62.66:43883)
0926 23:21:21.591890 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.591896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.592237 (+ 341us) server_negotiation.cc:407] Connection header received
0926 23:21:21.592416 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.592420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.592531 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.592641 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:21.593093 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.594002 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.596302 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.597521 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.597677 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.597749 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.598205 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.598334 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:21:21.598582 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.590711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:45573 (local address 127.16.62.67:41859)
0926 23:21:21.590929 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.590933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.591445 (+ 512us) server_negotiation.cc:407] Connection header received
0926 23:21:21.591655 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.591661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.591744 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.591850 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:21.592563 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.593668 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.596157 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.597457 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.597694 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.597756 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.598205 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.598334 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 23:21:21.598873 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.590617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:41859 (local address 127.16.62.65:45573)
0926 23:21:21.591332 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:21.591354 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:21.591494 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:21.591977 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:21.591982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:21.592010 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:21.592416 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.592425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.593788 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.593791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.595938 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.595953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.597817 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.597822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.598003 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.598061 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 23:21:21.598158 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 23:21:21.598389 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":88,"thread_start_us":404,"threads_started":1}
I20260926 23:21:21.598896 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.591467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:43883 (local address 127.16.62.65:41185)
0926 23:21:21.592099 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:21.592119 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:21.592266 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:21.592657 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:21.592659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:21.592678 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:21.592940 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.592947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.594116 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.594121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.596113 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:21.596122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.597796 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.597806 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:21.598036 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.598085 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 23:21:21.598158 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 23:21:21.598389 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":100,"thread_start_us":304,"threads_started":1}
I20260926 23:21:21.600080 2674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72530cb49613415e8c68be57cbcd2ef3" candidate_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a84ff9e6b8483390055e471a59b5c1" is_pre_election: true
I20260926 23:21:21.600080 2807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72530cb49613415e8c68be57cbcd2ef3" candidate_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c98bb4f4c44fd49c2135115e1958bb" is_pre_election: true
I20260926 23:21:21.600715 2674 raft_consensus.cc:2454] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c29bea331c1c4b89ae7ac4bd4a4b3180 in term 0.
I20260926 23:21:21.600692 2807 raft_consensus.cc:2454] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c29bea331c1c4b89ae7ac4bd4a4b3180 in term 0.
I20260926 23:21:21.601357 2473 leader_election.cc:302] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c98bb4f4c44fd49c2135115e1958bb, c29bea331c1c4b89ae7ac4bd4a4b3180; no voters:
I20260926 23:21:21.601670 2877 raft_consensus.cc:2781] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:21:21.601823 2877 raft_consensus.cc:482] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:21:21.601951 2877 raft_consensus.cc:3037] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:21.604502 2877 raft_consensus.cc:504] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } }
I20260926 23:21:21.604998 2877 leader_election.cc:288] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [CANDIDATE]: Term 1 election: Requested vote from peers 49c98bb4f4c44fd49c2135115e1958bb (127.16.62.67:41859), a5a84ff9e6b8483390055e471a59b5c1 (127.16.62.66:43883)
I20260926 23:21:21.605545 2807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72530cb49613415e8c68be57cbcd2ef3" candidate_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c98bb4f4c44fd49c2135115e1958bb"
I20260926 23:21:21.605593 2674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72530cb49613415e8c68be57cbcd2ef3" candidate_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a84ff9e6b8483390055e471a59b5c1"
I20260926 23:21:21.605768 2674 raft_consensus.cc:3037] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:21.605763 2807 raft_consensus.cc:3037] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:21.608968 2674 raft_consensus.cc:2454] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c29bea331c1c4b89ae7ac4bd4a4b3180 in term 1.
I20260926 23:21:21.608981 2807 raft_consensus.cc:2454] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c29bea331c1c4b89ae7ac4bd4a4b3180 in term 1.
I20260926 23:21:21.609316 2473 leader_election.cc:302] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c98bb4f4c44fd49c2135115e1958bb, c29bea331c1c4b89ae7ac4bd4a4b3180; no voters:
I20260926 23:21:21.609582 2877 raft_consensus.cc:2781] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:21.609977 2877 raft_consensus.cc:686] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 1 LEADER]: Becoming Leader. State: Replica: c29bea331c1c4b89ae7ac4bd4a4b3180, State: Running, Role: LEADER
I20260926 23:21:21.610445 2877 consensus_queue.cc:237] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } }
I20260926 23:21:21.615018 2396 catalog_manager.cc:5495] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 reported cstate change: term changed from 0 to 1, leader changed from <none> to c29bea331c1c4b89ae7ac4bd4a4b3180 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c29bea331c1c4b89ae7ac4bd4a4b3180" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 41609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 } health_report { overall_health: UNKNOWN } } }
W20260926 23:21:21.619628 2586 tablet.cc:2369] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:21:21.631009 2853 tablet.cc:2369] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:21:21.632447 2720 tablet.cc:2369] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:21:21.651185 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.645837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.16.62.65:41609)
0926 23:21:21.646017 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.646021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.646038 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:21:21.646201 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.646204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.646275 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.646391 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:21:21.646749 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.647488 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.648882 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.649357 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.649447 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.649902 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:21:21.650493 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:21:21.650708 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.650802 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.650865 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:21:21.658931 2807 raft_consensus.cc:1261] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [term 1 FOLLOWER]: Refusing update from remote peer c29bea331c1c4b89ae7ac4bd4a4b3180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:21.658931 2674 raft_consensus.cc:1261] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [term 1 FOLLOWER]: Refusing update from remote peer c29bea331c1c4b89ae7ac4bd4a4b3180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:21.659832 2881 consensus_queue.cc:1035] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5a84ff9e6b8483390055e471a59b5c1" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:21.660203 2877 consensus_queue.cc:1035] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c98bb4f4c44fd49c2135115e1958bb" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 41859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:21.670179 2884 mvcc.cc:205] Tried to move back new op lower bound from 7333744155264876544 to 7333744155080433664. Current Snapshot: MvccSnapshot[applied={T|T < 7333744155264876544}]
I20260926 23:21:21.672071 2886 mvcc.cc:205] Tried to move back new op lower bound from 7333744155264876544 to 7333744155080433664. Current Snapshot: MvccSnapshot[applied={T|T < 7333744155264876544}]
I20260926 23:21:21.682430 2396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:21.682668 2396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:21.685818 2396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 162af647614e489a9feb7c23490c60fd: Sending DeleteTablet for 3 replicas of tablet 72530cb49613415e8c68be57cbcd2ef3
I20260926 23:21:21.686795 2787 tablet_service.cc:1514] Processing DeleteTablet for tablet 72530cb49613415e8c68be57cbcd2ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:21 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:35390
I20260926 23:21:21.686939 2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 72530cb49613415e8c68be57cbcd2ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:21 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:46014
I20260926 23:21:21.687139 2654 tablet_service.cc:1514] Processing DeleteTablet for tablet 72530cb49613415e8c68be57cbcd2ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:21 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:47296
I20260926 23:21:21.692031 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 2457
I20260926 23:21:21.700258 2895 tablet_replica.cc:330] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: stopping tablet replica
I20260926 23:21:21.700902 2895 raft_consensus.cc:2229] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:21.701583 2895 raft_consensus.cc:2258] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:21.704128 2895 ts_tablet_manager.cc:1905] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:21.712741 2894 tablet_replica.cc:330] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: stopping tablet replica
I20260926 23:21:21.713277 2894 raft_consensus.cc:2229] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:21:21.713428 2895 ts_tablet_manager.cc:1918] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:21.713645 2895 log.cc:1196] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/72530cb49613415e8c68be57cbcd2ef3
I20260926 23:21:21.714206 2895 ts_tablet_manager.cc:1939] T 72530cb49613415e8c68be57cbcd2ef3 P a5a84ff9e6b8483390055e471a59b5c1: Deleting consensus metadata
I20260926 23:21:21.714157 2894 raft_consensus.cc:2258] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:21.716630 2380 catalog_manager.cc:4841] TS a5a84ff9e6b8483390055e471a59b5c1 (127.16.62.66:43883): tablet 72530cb49613415e8c68be57cbcd2ef3 (table test-table [id=92cdb4026f89410f8310ecd8e82b0a4a]) successfully deleted
I20260926 23:21:21.716852 2894 ts_tablet_manager.cc:1905] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:21.718238 2893 tablet_replica.cc:330] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: stopping tablet replica
I20260926 23:21:21.718786 2893 raft_consensus.cc:2229] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:21.719386 2893 raft_consensus.cc:2258] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:21.725114 2894 ts_tablet_manager.cc:1918] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:21.725286 2894 log.cc:1196] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/72530cb49613415e8c68be57cbcd2ef3
I20260926 23:21:21.725327 2893 ts_tablet_manager.cc:1905] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 23:21:21 dist-test-slave-vcv1 krb5kdc[2351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:21:21.725844 2894 ts_tablet_manager.cc:1939] T 72530cb49613415e8c68be57cbcd2ef3 P c29bea331c1c4b89ae7ac4bd4a4b3180: Deleting consensus metadata
I20260926 23:21:21.727666 2383 catalog_manager.cc:4841] TS c29bea331c1c4b89ae7ac4bd4a4b3180 (127.16.62.65:41609): tablet 72530cb49613415e8c68be57cbcd2ef3 (table test-table [id=92cdb4026f89410f8310ecd8e82b0a4a]) successfully deleted
I20260926 23:21:21.732755 2893 ts_tablet_manager.cc:1918] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:21.732985 2893 log.cc:1196] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/72530cb49613415e8c68be57cbcd2ef3
I20260926 23:21:21.733397 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.693878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.16.62.65:41609)
0926 23:21:21.694096 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.694105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.712809 (+ 18704us) server_negotiation.cc:407] Connection header received
0926 23:21:21.713294 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.713303 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.713405 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.713478 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:21.715506 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.716705 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.718729 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.719465 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.719618 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.727015 (+ 7397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:21.727027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:21.727044 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:21.727084 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:21.730355 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.731037 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.731046 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.731054 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.731199 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.731594 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.731599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.731602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.731932 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:21.732331 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.732565 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.732810 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:21:21.733561 2893 ts_tablet_manager.cc:1939] T 72530cb49613415e8c68be57cbcd2ef3 P 49c98bb4f4c44fd49c2135115e1958bb: Deleting consensus metadata
I20260926 23:21:21.734347 2580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:21.735301 2380 catalog_manager.cc:4841] TS 49c98bb4f4c44fd49c2135115e1958bb (127.16.62.67:41859): tablet 72530cb49613415e8c68be57cbcd2ef3 (table test-table [id=92cdb4026f89410f8310ecd8e82b0a4a]) successfully deleted
I20260926 23:21:21.836859 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 2457
I20260926 23:21:21.851388 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 2591
Sep 26 23:21:21 dist-test-slave-vcv1 krb5kdc[2351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:21:21.866621 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.852256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.16.62.66:43883)
0926 23:21:21.852441 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.852447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.852467 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:21:21.852604 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.852607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.852668 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.852773 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:21.854058 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.854855 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.856110 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.856629 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.856763 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.860955 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:21.860967 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:21.860981 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:21.861020 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:21.863819 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.864571 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.864578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.864580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.864707 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.865083 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.865094 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.865096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.865445 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:21.865753 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.866061 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.866289 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:21:21.867331 2714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:21.956092 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 2591
I20260926 23:21:21.972949 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 2724
Sep 26 23:21:21 dist-test-slave-vcv1 krb5kdc[2351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:21:21.987387 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:21.973780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.16.62.67:41859)
0926 23:21:21.973949 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:21.973954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:21.974006 (+ 52us) server_negotiation.cc:407] Connection header received
0926 23:21:21.974190 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:21.974194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:21.974257 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:21.974349 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:21.975662 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.976463 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.977711 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:21.978093 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:21.978191 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:21.982547 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:21.982561 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:21.982577 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:21.982607 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:21.985176 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.985773 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.985779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.985782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.985902 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:21.986209 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:21.986212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:21.986214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:21.986535 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:21.986736 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:21.986981 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 23:21:21.987154 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:21:21.988018 2847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:22.075704 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 2724
I20260926 23:21:22.092417 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 2364
I20260926 23:21:22.093413 2426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:22.203500 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 2364
2026-09-26T23:21:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4196 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T23:21:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:22Z Disabled control of system clock
I20260926 23:21:22.279059 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41729
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:41729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:21:22.618682 2904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:22.618918 2904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:22.618992 2904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:22.625891 2904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:22.625985 2904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:22.626021 2904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:22.626068 2904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:22.626103 2904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:22.635128 2904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:41729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:41729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:22.636328 2904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:22.638053 2904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:22.649632 2912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:22.649734 2909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:22.649734 2910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:22.650646 2904 server_base.cc:1034] running on GCE node
I20260926 23:21:22.651388 2904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:22.652770 2904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:22.654001 2904 hybrid_clock.cc:648] HybridClock initialized: now 1790464882653962 us; error 43 us; skew 500 ppm
I20260926 23:21:22.654345 2904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:22.656543 2904 webserver.cc:460] Webserver started at http://127.16.62.126:44633/ using document root <none> and password file <none>
I20260926 23:21:22.657191 2904 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:22.657320 2904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:22.657631 2904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:22.660523 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "4d7e07f0166942baa9cfd37353f1c0be"
format_stamp: "Formatted at 2026-09-26 23:21:22 on dist-test-slave-vcv1"
server_key: "7a608f01c8b5ff1f6a9f1033e3bac873"
server_key_iv: "a628f87d53ac65f6c406763081a0b7a3"
server_key_version: "encryptionkey@0"
I20260926 23:21:22.661329 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "4d7e07f0166942baa9cfd37353f1c0be"
format_stamp: "Formatted at 2026-09-26 23:21:22 on dist-test-slave-vcv1"
server_key: "7a608f01c8b5ff1f6a9f1033e3bac873"
server_key_iv: "a628f87d53ac65f6c406763081a0b7a3"
server_key_version: "encryptionkey@0"
I20260926 23:21:22.666169 2904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:21:22.669479 2918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:22.670683 2904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:21:22.670866 2904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "4d7e07f0166942baa9cfd37353f1c0be"
format_stamp: "Formatted at 2026-09-26 23:21:22 on dist-test-slave-vcv1"
server_key: "7a608f01c8b5ff1f6a9f1033e3bac873"
server_key_iv: "a628f87d53ac65f6c406763081a0b7a3"
server_key_version: "encryptionkey@0"
I20260926 23:21:22.671072 2904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:22.693524 2904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:22.694308 2904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:22.694592 2904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:22.713182 2904 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:41729
I20260926 23:21:22.713200 2969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:41729 every 8 connection(s)
I20260926 23:21:22.714403 2904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:22.717896 2970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:22.719511 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 2904
I20260926 23:21:22.719692 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:22.720099 16633 external_mini_cluster.cc:1428] Setting key 504aa52be29fd53540b53a19c990e259
I20260926 23:21:22.730747 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be: Bootstrap starting.
I20260926 23:21:22.734443 2970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:22.735347 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:22.722001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.16.62.126:41729)
0926 23:21:22.723045 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:22.723058 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:22.723111 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:21:22.725419 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:22.725445 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:22.725993 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:22.726428 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:22.726970 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:22.729024 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:22.730385 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:22.731388 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:22.731551 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:22.732657 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:22.732667 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:22.732683 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:22.733348 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:22.733462 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:22.733491 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:22.733613 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:22.733800 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 23:21:22.734076 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":341,"threads_started":1}
I20260926 23:21:22.735759 2970 log.cc:824] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:22.738785 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be: No bootstrap required, opened a new log
I20260926 23:21:22.746083 2970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7e07f0166942baa9cfd37353f1c0be" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41729 } }
I20260926 23:21:22.746429 2970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:22.746531 2970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7e07f0166942baa9cfd37353f1c0be, State: Initialized, Role: FOLLOWER
I20260926 23:21:22.747094 2970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7e07f0166942baa9cfd37353f1c0be" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41729 } }
I20260926 23:21:22.747332 2970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:22.747467 2970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:22.747648 2970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:22.749761 2970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7e07f0166942baa9cfd37353f1c0be" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41729 } }
I20260926 23:21:22.750218 2970 leader_election.cc:302] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d7e07f0166942baa9cfd37353f1c0be; no voters:
I20260926 23:21:22.750756 2970 leader_election.cc:288] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:22.750908 2975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:22.752444 2975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [term 1 LEADER]: Becoming Leader. State: Replica: 4d7e07f0166942baa9cfd37353f1c0be, State: Running, Role: LEADER
I20260926 23:21:22.752918 2975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7e07f0166942baa9cfd37353f1c0be" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41729 } }
I20260926 23:21:22.753052 2970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:22.755735 2977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d7e07f0166942baa9cfd37353f1c0be. Latest consensus state: current_term: 1 leader_uuid: "4d7e07f0166942baa9cfd37353f1c0be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7e07f0166942baa9cfd37353f1c0be" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41729 } } }
I20260926 23:21:22.756197 2977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:22.756350 2976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d7e07f0166942baa9cfd37353f1c0be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7e07f0166942baa9cfd37353f1c0be" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 41729 } } }
I20260926 23:21:22.756691 2976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:22.761577 2983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:22.766383 2983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:22.774350 2983 catalog_manager.cc:1347] Generated new cluster ID: e93b6abb8f854cf3a0316abba9cbf01d
I20260926 23:21:22.774448 2983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:22.806572 2983 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:22.807844 2983 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:22.812958 2983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be: Generated new TSK 0
I20260926 23:21:22.813730 2983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:22.819929 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41729
--builtin_ntp_servers=127.16.62.84:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:21:23.140295 2994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:23.140547 2994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:23.140708 2994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:23.147883 2994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:23.147969 2994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:23.148096 2994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:21:23.157058 2994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:23.158186 2994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:23.159811 2994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:23.172209 3000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:23.172477 2999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:23.172683 3002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:23.173265 2994 server_base.cc:1034] running on GCE node
I20260926 23:21:23.173852 2994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:23.174783 2994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:23.175981 2994 hybrid_clock.cc:648] HybridClock initialized: now 1790464883175910 us; error 59 us; skew 500 ppm
I20260926 23:21:23.176342 2994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:23.178453 2994 webserver.cc:460] Webserver started at http://127.16.62.65:35415/ using document root <none> and password file <none>
I20260926 23:21:23.179087 2994 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:23.179205 2994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:23.179518 2994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:23.182288 2994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "327aeb357a6046b3816704d25773e576"
format_stamp: "Formatted at 2026-09-26 23:21:23 on dist-test-slave-vcv1"
server_key: "32dd2d587d7e35b641f06e1610512c15"
server_key_iv: "4439b1baa538b637c8437d99a2555640"
server_key_version: "encryptionkey@0"
I20260926 23:21:23.183117 2994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "327aeb357a6046b3816704d25773e576"
format_stamp: "Formatted at 2026-09-26 23:21:23 on dist-test-slave-vcv1"
server_key: "32dd2d587d7e35b641f06e1610512c15"
server_key_iv: "4439b1baa538b637c8437d99a2555640"
server_key_version: "encryptionkey@0"
I20260926 23:21:23.187889 2994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 23:21:23.191184 3008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:23.192454 2994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 23:21:23.192613 2994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "327aeb357a6046b3816704d25773e576"
format_stamp: "Formatted at 2026-09-26 23:21:23 on dist-test-slave-vcv1"
server_key: "32dd2d587d7e35b641f06e1610512c15"
server_key_iv: "4439b1baa538b637c8437d99a2555640"
server_key_version: "encryptionkey@0"
I20260926 23:21:23.192878 2994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:23.204999 2994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:23.205726 2994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:23.206013 2994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:23.206900 2994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:23.208290 2994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:23.208403 2994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:23.208541 2994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:23.208630 2994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:23.240687 2994 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:37133
I20260926 23:21:23.240774 3120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:37133 every 8 connection(s)
I20260926 23:21:23.241969 2994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:21:23.243700 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 2994
I20260926 23:21:23.243922 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:21:23.244473 16633 external_mini_cluster.cc:1428] Setting key 18f7077257541f9c6bda443c3a7b063f
I20260926 23:21:23.247671 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41729
--builtin_ntp_servers=127.16.62.84:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:21:23.256940 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:23.244514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:46669 (local address 127.16.62.126:41729)
0926 23:21:23.244985 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:23.244991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:23.246491 (+ 1500us) server_negotiation.cc:407] Connection header received
0926 23:21:23.249131 (+ 2640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:23.249135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:23.249199 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:23.249289 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:23.250612 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:23.251307 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:23.253835 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:23.254321 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:23.254412 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:23.255960 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:23.255965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:23.255968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:23.255984 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:23.256034 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:23.256046 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:23.256114 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:23.256414 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 23:21:23.256529 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":242,"threads_started":1}
I20260926 23:21:23.257849 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:23.245232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41729 (local address 127.16.62.65:46669)
0926 23:21:23.246320 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:23.246374 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:23.248844 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:23.249545 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:23.249557 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:23.249679 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:23.250451 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:23.250477 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:23.251459 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:23.251467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:23.253705 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:23.253723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:23.254533 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:23.254542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:23.254878 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:23.255607 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:21:23.255613 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:23.255684 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:21:23.255781 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:23.256166 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:23.256180 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:23.256192 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 23:21:23.256377 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 23:21:23.256698 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":309,"threads_started":1}
I20260926 23:21:23.259349 3121 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41729
I20260926 23:21:23.259663 3121 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:23.260528 3121 heartbeater.cc:511] Master 127.16.62.126:41729 requested a full tablet report, sending...
I20260926 23:21:23.262696 2935 ts_manager.cc:194] Registered new tserver with Master: 327aeb357a6046b3816704d25773e576 (127.16.62.65:37133)
I20260926 23:21:23.264219 2935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.65:46669
W20260926 23:21:23.573951 3125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:23.574265 3125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:23.574385 3125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:23.581523 3125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:23.581616 3125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:23.581718 3125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:21:23.590425 3125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:23.591589 3125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:23.593317 3125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:23.605665 3130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:23.606112 3131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:23.606966 3133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:23.608072 3125 server_base.cc:1034] running on GCE node
I20260926 23:21:23.608771 3125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:23.609692 3125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:23.610917 3125 hybrid_clock.cc:648] HybridClock initialized: now 1790464883610872 us; error 27 us; skew 500 ppm
I20260926 23:21:23.611227 3125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:23.613415 3125 webserver.cc:460] Webserver started at http://127.16.62.66:44611/ using document root <none> and password file <none>
I20260926 23:21:23.614094 3125 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:23.614214 3125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:23.614543 3125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:23.617486 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "cfd3774765a34d31a8c866b8f3828d9b"
format_stamp: "Formatted at 2026-09-26 23:21:23 on dist-test-slave-vcv1"
server_key: "734d789ad6570f29371b424149a16ceb"
server_key_iv: "c008d0fcee5eca46bfb5ae40743c8575"
server_key_version: "encryptionkey@0"
I20260926 23:21:23.618287 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "cfd3774765a34d31a8c866b8f3828d9b"
format_stamp: "Formatted at 2026-09-26 23:21:23 on dist-test-slave-vcv1"
server_key: "734d789ad6570f29371b424149a16ceb"
server_key_iv: "c008d0fcee5eca46bfb5ae40743c8575"
server_key_version: "encryptionkey@0"
I20260926 23:21:23.623268 3125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:21:23.626461 3139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:23.627683 3125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:21:23.627851 3125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "cfd3774765a34d31a8c866b8f3828d9b"
format_stamp: "Formatted at 2026-09-26 23:21:23 on dist-test-slave-vcv1"
server_key: "734d789ad6570f29371b424149a16ceb"
server_key_iv: "c008d0fcee5eca46bfb5ae40743c8575"
server_key_version: "encryptionkey@0"
I20260926 23:21:23.628054 3125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:23.642115 3125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:23.642890 3125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:23.643190 3125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:23.644075 3125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:23.645498 3125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:23.645643 3125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:23.645772 3125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:23.645859 3125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:23.680763 3125 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:42259
I20260926 23:21:23.680816 3251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:42259 every 8 connection(s)
I20260926 23:21:23.682097 3125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:21:23.685801 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 3125
I20260926 23:21:23.685981 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:21:23.686488 16633 external_mini_cluster.cc:1428] Setting key 596752b0fc7d25031d31686b638b46c1
I20260926 23:21:23.689505 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41729
--builtin_ntp_servers=127.16.62.84:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:21:23.695804 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:23.684624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:55317 (local address 127.16.62.126:41729)
0926 23:21:23.684802 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:23.684807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:23.686301 (+ 1494us) server_negotiation.cc:407] Connection header received
0926 23:21:23.688589 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:23.688594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:23.688682 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:23.688787 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:23.689894 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:23.691103 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:23.692733 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:23.693442 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:23.693584 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:23.694862 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:23.694870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:23.694878 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:23.694904 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:23.694945 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:23.694964 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:23.695058 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:23.695256 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 23:21:23.695367 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:21:23.696720 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:23.685155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41729 (local address 127.16.62.66:55317)
0926 23:21:23.686152 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:23.686194 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:23.688250 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:23.688979 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:23.688987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:23.689074 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:23.689731 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:23.689755 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:23.691255 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:23.691259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:23.692588 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:23.692599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:23.693650 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:23.693654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:23.693845 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:23.694550 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:21:23.694553 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:23.694590 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:21:23.694675 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:23.695059 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:23.695066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:23.695071 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:21:23.695214 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 23:21:23.695510 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":318,"threads_started":1}
I20260926 23:21:23.698201 3252 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41729
I20260926 23:21:23.698513 3252 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:23.699266 3252 heartbeater.cc:511] Master 127.16.62.126:41729 requested a full tablet report, sending...
I20260926 23:21:23.700718 2935 ts_manager.cc:194] Registered new tserver with Master: cfd3774765a34d31a8c866b8f3828d9b (127.16.62.66:42259)
I20260926 23:21:23.702107 2935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.66:55317
W20260926 23:21:24.019209 3255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:24.019472 3255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:24.019557 3255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:24.026604 3255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:24.026695 3255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:24.026795 3255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:21:24.035710 3255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:41729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:24.036927 3255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:24.038543 3255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:24.050961 3260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:24.050997 3261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:24.051029 3263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:24.051808 3255 server_base.cc:1034] running on GCE node
I20260926 23:21:24.052472 3255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:24.053359 3255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:24.054589 3255 hybrid_clock.cc:648] HybridClock initialized: now 1790464884054533 us; error 45 us; skew 500 ppm
I20260926 23:21:24.054847 3255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:24.056900 3255 webserver.cc:460] Webserver started at http://127.16.62.67:34351/ using document root <none> and password file <none>
I20260926 23:21:24.057480 3255 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:24.057560 3255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:24.057801 3255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:24.060495 3255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "d9ae385097d247a38a14e34b8c9627d3"
format_stamp: "Formatted at 2026-09-26 23:21:24 on dist-test-slave-vcv1"
server_key: "c383e4fae9555298437f854de3ee7c25"
server_key_iv: "176e12f55acf88e27a87eaff9bcf8421"
server_key_version: "encryptionkey@0"
I20260926 23:21:24.061189 3255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "d9ae385097d247a38a14e34b8c9627d3"
format_stamp: "Formatted at 2026-09-26 23:21:24 on dist-test-slave-vcv1"
server_key: "c383e4fae9555298437f854de3ee7c25"
server_key_iv: "176e12f55acf88e27a87eaff9bcf8421"
server_key_version: "encryptionkey@0"
I20260926 23:21:24.065793 3255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:21:24.068993 3269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:24.070340 3255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:24.070544 3255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "d9ae385097d247a38a14e34b8c9627d3"
format_stamp: "Formatted at 2026-09-26 23:21:24 on dist-test-slave-vcv1"
server_key: "c383e4fae9555298437f854de3ee7c25"
server_key_iv: "176e12f55acf88e27a87eaff9bcf8421"
server_key_version: "encryptionkey@0"
I20260926 23:21:24.070775 3255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:24.087881 3255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:24.088672 3255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:24.088981 3255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:24.089875 3255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:24.091228 3255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:24.091348 3255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:24.091473 3255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:24.091559 3255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:24.123811 3255 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:45251
I20260926 23:21:24.123864 3381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:45251 every 8 connection(s)
I20260926 23:21:24.125128 3255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:21:24.125370 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 3255
I20260926 23:21:24.125525 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:21:24.126155 16633 external_mini_cluster.cc:1428] Setting key e9a9ced0c37f78b26955af67c9c4560f
I20260926 23:21:24.137532 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.128055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:36167 (local address 127.16.62.126:41729)
0926 23:21:24.128304 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.128310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.129902 (+ 1592us) server_negotiation.cc:407] Connection header received
0926 23:21:24.131703 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.131707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.131768 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.131848 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:24.132947 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.133690 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.135200 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.135568 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.135664 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.136908 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:24.136912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:24.136916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:24.136931 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:24.136962 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:24.136973 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:24.137041 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.137235 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.137304 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:21:24.138453 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.128736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:41729 (local address 127.16.62.67:36167)
0926 23:21:24.129748 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.129792 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:24.131478 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:24.132064 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:24.132072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:24.132207 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:24.132802 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.132818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.133844 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.133847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.135085 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.135096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.135762 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.135767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.135976 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.136635 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:21:24.136638 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:24.136673 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:21:24.136745 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:24.137056 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:24.137064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:24.137069 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:21:24.137194 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 23:21:24.137458 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":332,"threads_started":1}
I20260926 23:21:24.139838 3382 heartbeater.cc:347] Connected to a master server at 127.16.62.126:41729
I20260926 23:21:24.140233 3382 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:24.141027 3382 heartbeater.cc:511] Master 127.16.62.126:41729 requested a full tablet report, sending...
I20260926 23:21:24.142339 2935 ts_manager.cc:194] Registered new tserver with Master: d9ae385097d247a38a14e34b8c9627d3 (127.16.62.67:45251)
I20260926 23:21:24.143055 2935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.67:36167
I20260926 23:21:24.153461 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:21:24.163077 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.157497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.16.62.126:41729)
0926 23:21:24.157745 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.157754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.158061 (+ 307us) server_negotiation.cc:407] Connection header received
0926 23:21:24.158182 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.158186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.158252 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.158329 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:24.158796 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.159440 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.160877 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.161269 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.161355 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.162386 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:24.162391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:24.162394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:24.162410 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:24.162438 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:24.162449 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:24.162551 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.162712 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.162781 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:21:24.166437 2935 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:21:24.173110 2935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:21:24.207623 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.184763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:42259 (local address 127.0.0.1:46918)
0926 23:21:24.198626 (+ 13863us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.198673 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:24.198835 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:24.200863 (+ 2028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:24.200871 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:24.200910 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:24.201294 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.201308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.202949 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.202955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.205169 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.205181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.206714 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.206720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.206902 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.206951 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 23:21:24.207065 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 23:21:24.207260 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13606,"mutex_wait_us":65,"thread_start_us":15552,"threads_started":1}
I20260926 23:21:24.208688 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.185092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.16.62.66:42259)
0926 23:21:24.200102 (+ 15010us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.200114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.200184 (+ 70us) server_negotiation.cc:407] Connection header received
0926 23:21:24.200315 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.200326 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.200621 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.201448 (+ 827us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:24.201588 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.202825 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.205340 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.206535 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.206674 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.208079 (+ 1405us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.208194 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.208363 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14804,"thread_start_us":14671,"threads_started":1}
I20260926 23:21:24.209120 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.185093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37133 (local address 127.0.0.1:55040)
0926 23:21:24.199545 (+ 14452us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.199564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:24.200609 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:24.201551 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:24.201555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:24.201578 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:24.201933 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.201945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.203775 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.203780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.206130 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.206144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.208237 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.208243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.208421 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.208465 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 23:21:24.208570 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 23:21:24.208718 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14341,"mutex_wait_us":3,"thread_start_us":13125,"threads_started":1}
I20260926 23:21:24.209362 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.185629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.16.62.65:37133)
0926 23:21:24.197824 (+ 12195us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.197837 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.199663 (+ 1826us) server_negotiation.cc:407] Connection header received
0926 23:21:24.200859 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.200870 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.201096 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.201272 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:24.202150 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.203602 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.206327 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.207721 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.207855 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.208079 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.208640 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.208925 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11970,"thread_start_us":12196,"threads_started":1}
I20260926 23:21:24.209709 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.199646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.16.62.67:45251)
0926 23:21:24.200465 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.200475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.201091 (+ 616us) server_negotiation.cc:407] Connection header received
0926 23:21:24.201541 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.201548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.201762 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.201963 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:24.202554 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.203996 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.206529 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.207928 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.208045 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.208327 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.209027 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.209321 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":495,"threads_started":1}
I20260926 23:21:24.210071 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.199485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:45251 (local address 127.0.0.1:35762)
0926 23:21:24.200836 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.200854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:24.200965 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:24.201978 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:24.201983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:24.202013 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:24.202363 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.202373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.204181 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.204186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.206370 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.206380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.208425 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.208429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.208596 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.208628 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:21:24.208720 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:21:24.208834 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1234,"mutex_wait_us":3,"thread_start_us":1105,"threads_started":1}
I20260926 23:21:24.212549 3187 tablet_service.cc:1467] Processing CreateTablet for tablet 9615a708166340d8bdc6e7dc90e1223b (DEFAULT_TABLE table=test-table [id=c90c3a0ea3264e7183ecd38eb6691509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:24.212548 3056 tablet_service.cc:1467] Processing CreateTablet for tablet 9615a708166340d8bdc6e7dc90e1223b (DEFAULT_TABLE table=test-table [id=c90c3a0ea3264e7183ecd38eb6691509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:24.213339 3317 tablet_service.cc:1467] Processing CreateTablet for tablet 9615a708166340d8bdc6e7dc90e1223b (DEFAULT_TABLE table=test-table [id=c90c3a0ea3264e7183ecd38eb6691509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:24.213830 3056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9615a708166340d8bdc6e7dc90e1223b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:24.213951 3187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9615a708166340d8bdc6e7dc90e1223b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:24.214548 3317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9615a708166340d8bdc6e7dc90e1223b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:24.225054 3400 tablet_bootstrap.cc:492] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576: Bootstrap starting.
I20260926 23:21:24.227890 3401 tablet_bootstrap.cc:492] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: Bootstrap starting.
I20260926 23:21:24.228446 3400 tablet_bootstrap.cc:654] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:24.229477 3402 tablet_bootstrap.cc:492] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: Bootstrap starting.
I20260926 23:21:24.229861 3400 log.cc:824] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:24.231364 3401 tablet_bootstrap.cc:654] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:24.232560 3402 tablet_bootstrap.cc:654] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:24.232659 3401 log.cc:824] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:24.233783 3400 tablet_bootstrap.cc:492] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576: No bootstrap required, opened a new log
I20260926 23:21:24.233784 3402 log.cc:824] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:24.234071 3400 ts_tablet_manager.cc:1397] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260926 23:21:24.235527 3401 tablet_bootstrap.cc:492] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: No bootstrap required, opened a new log
I20260926 23:21:24.235791 3401 ts_tablet_manager.cc:1397] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260926 23:21:24.237655 3402 tablet_bootstrap.cc:492] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: No bootstrap required, opened a new log
I20260926 23:21:24.237895 3402 ts_tablet_manager.cc:1397] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 23:21:24.246193 3400 raft_consensus.cc:348] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } }
I20260926 23:21:24.246206 3401 raft_consensus.cc:348] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } }
I20260926 23:21:24.246392 3402 raft_consensus.cc:348] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } }
I20260926 23:21:24.246675 3401 raft_consensus.cc:374] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:24.246773 3402 raft_consensus.cc:374] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:24.246788 3401 raft_consensus.cc:729] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd3774765a34d31a8c866b8f3828d9b, State: Initialized, Role: FOLLOWER
I20260926 23:21:24.246877 3402 raft_consensus.cc:729] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ae385097d247a38a14e34b8c9627d3, State: Initialized, Role: FOLLOWER
I20260926 23:21:24.246922 3400 raft_consensus.cc:374] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:24.247105 3400 raft_consensus.cc:729] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327aeb357a6046b3816704d25773e576, State: Initialized, Role: FOLLOWER
I20260926 23:21:24.247568 3402 consensus_queue.cc:260] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } }
I20260926 23:21:24.247604 3401 consensus_queue.cc:260] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } }
I20260926 23:21:24.247699 3400 consensus_queue.cc:260] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } }
I20260926 23:21:24.255626 3401 ts_tablet_manager.cc:1428] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: Time spent starting tablet: real 0.020s user 0.007s sys 0.011s
I20260926 23:21:24.256834 3252 heartbeater.cc:503] Master 127.16.62.126:41729 was elected leader, sending a full tablet report...
I20260926 23:21:24.257156 3402 ts_tablet_manager.cc:1428] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: Time spent starting tablet: real 0.019s user 0.009s sys 0.011s
I20260926 23:21:24.257507 3400 ts_tablet_manager.cc:1428] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576: Time spent starting tablet: real 0.023s user 0.006s sys 0.015s
I20260926 23:21:24.258064 3382 heartbeater.cc:503] Master 127.16.62.126:41729 was elected leader, sending a full tablet report...
I20260926 23:21:24.258167 3121 heartbeater.cc:503] Master 127.16.62.126:41729 was elected leader, sending a full tablet report...
I20260926 23:21:24.280006 3408 raft_consensus.cc:482] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:21:24.280332 3408 raft_consensus.cc:504] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } }
I20260926 23:21:24.282019 3408 leader_election.cc:288] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfd3774765a34d31a8c866b8f3828d9b (127.16.62.66:42259), d9ae385097d247a38a14e34b8c9627d3 (127.16.62.67:45251)
I20260926 23:21:24.290911 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.282276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:36053 (local address 127.16.62.66:42259)
0926 23:21:24.282479 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.282484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.283437 (+ 953us) server_negotiation.cc:407] Connection header received
0926 23:21:24.283650 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.283670 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.283763 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.283866 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:24.284578 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.285687 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.288698 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.289870 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.290030 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.290100 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.290538 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.290674 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:21:24.290917 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.282567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:43339 (local address 127.16.62.67:45251)
0926 23:21:24.282794 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.282802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.283587 (+ 785us) server_negotiation.cc:407] Connection header received
0926 23:21:24.283728 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.283733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.283804 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.283889 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:24.284622 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.285687 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.288697 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.289900 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.290034 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.290100 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.290538 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.290662 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:21:24.291133 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.282457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:45251 (local address 127.16.62.65:43339)
0926 23:21:24.283486 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.283500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:24.283606 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:24.283914 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:24.283919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:24.284006 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:24.284461 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.284473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.285823 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.285828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.288474 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.288489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.290132 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.290139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.290352 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.290415 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 23:21:24.290501 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:21:24.290694 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":244,"spinlock_wait_cycles":21888,"thread_start_us":494,"threads_started":1}
I20260926 23:21:24.291426 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.282155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:42259 (local address 127.16.62.65:36053)
0926 23:21:24.283317 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.283339 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:24.283488 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:24.283914 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:24.283919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:24.283946 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:24.284415 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.284434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.285824 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.285828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.288475 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:24.288489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.290086 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.290096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:24.290358 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.290407 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 23:21:24.290501 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 23:21:24.290694 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":300,"thread_start_us":1051,"threads_started":1}
I20260926 23:21:24.292120 3337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9615a708166340d8bdc6e7dc90e1223b" candidate_uuid: "327aeb357a6046b3816704d25773e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ae385097d247a38a14e34b8c9627d3" is_pre_election: true
I20260926 23:21:24.292371 3207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9615a708166340d8bdc6e7dc90e1223b" candidate_uuid: "327aeb357a6046b3816704d25773e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd3774765a34d31a8c866b8f3828d9b" is_pre_election: true
I20260926 23:21:24.292694 3337 raft_consensus.cc:2454] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aeb357a6046b3816704d25773e576 in term 0.
I20260926 23:21:24.292868 3207 raft_consensus.cc:2454] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327aeb357a6046b3816704d25773e576 in term 0.
I20260926 23:21:24.293318 3010 leader_election.cc:302] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 327aeb357a6046b3816704d25773e576, d9ae385097d247a38a14e34b8c9627d3; no voters:
I20260926 23:21:24.293681 3408 raft_consensus.cc:2781] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:21:24.293828 3408 raft_consensus.cc:482] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:21:24.293938 3408 raft_consensus.cc:3037] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:24.296087 3408 raft_consensus.cc:504] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } }
I20260926 23:21:24.296692 3408 leader_election.cc:288] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [CANDIDATE]: Term 1 election: Requested vote from peers cfd3774765a34d31a8c866b8f3828d9b (127.16.62.66:42259), d9ae385097d247a38a14e34b8c9627d3 (127.16.62.67:45251)
I20260926 23:21:24.297088 3207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9615a708166340d8bdc6e7dc90e1223b" candidate_uuid: "327aeb357a6046b3816704d25773e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd3774765a34d31a8c866b8f3828d9b"
I20260926 23:21:24.297128 3337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9615a708166340d8bdc6e7dc90e1223b" candidate_uuid: "327aeb357a6046b3816704d25773e576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ae385097d247a38a14e34b8c9627d3"
I20260926 23:21:24.297274 3207 raft_consensus.cc:3037] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:24.297308 3337 raft_consensus.cc:3037] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:24.299904 3207 raft_consensus.cc:2454] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aeb357a6046b3816704d25773e576 in term 1.
I20260926 23:21:24.299906 3337 raft_consensus.cc:2454] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327aeb357a6046b3816704d25773e576 in term 1.
I20260926 23:21:24.300310 3011 leader_election.cc:302] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 327aeb357a6046b3816704d25773e576, cfd3774765a34d31a8c866b8f3828d9b; no voters:
I20260926 23:21:24.300657 3408 raft_consensus.cc:2781] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:24.301069 3408 raft_consensus.cc:686] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 1 LEADER]: Becoming Leader. State: Replica: 327aeb357a6046b3816704d25773e576, State: Running, Role: LEADER
I20260926 23:21:24.301508 3408 consensus_queue.cc:237] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } }
I20260926 23:21:24.306005 2933 catalog_manager.cc:5495] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 327aeb357a6046b3816704d25773e576 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "327aeb357a6046b3816704d25773e576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327aeb357a6046b3816704d25773e576" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 } health_report { overall_health: UNKNOWN } } }
I20260926 23:21:24.335529 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.329855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.16.62.65:37133)
0926 23:21:24.330074 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.330080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.330186 (+ 106us) server_negotiation.cc:407] Connection header received
0926 23:21:24.330427 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.330431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.330543 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.330713 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:21:24.331103 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.331820 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.333321 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.333735 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.333867 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.334470 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:21:24.334939 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:21:24.335125 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.335242 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.335285 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:21:24.343709 3337 raft_consensus.cc:1261] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [term 1 FOLLOWER]: Refusing update from remote peer 327aeb357a6046b3816704d25773e576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:24.343709 3207 raft_consensus.cc:1261] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [term 1 FOLLOWER]: Refusing update from remote peer 327aeb357a6046b3816704d25773e576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:24.344821 3411 consensus_queue.cc:1035] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9ae385097d247a38a14e34b8c9627d3" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 45251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:24.345219 3408 consensus_queue.cc:1035] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd3774765a34d31a8c866b8f3828d9b" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 42259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:24.377739 2933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:24.378039 2933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:24.381731 2933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d7e07f0166942baa9cfd37353f1c0be: Sending DeleteTablet for 3 replicas of tablet 9615a708166340d8bdc6e7dc90e1223b
I20260926 23:21:24.382776 3187 tablet_service.cc:1514] Processing DeleteTablet for tablet 9615a708166340d8bdc6e7dc90e1223b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:24 UTC) from {username='slave'} at 127.0.0.1:46918
I20260926 23:21:24.382850 3317 tablet_service.cc:1514] Processing DeleteTablet for tablet 9615a708166340d8bdc6e7dc90e1223b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:24 UTC) from {username='slave'} at 127.0.0.1:35762
I20260926 23:21:24.383113 3056 tablet_service.cc:1514] Processing DeleteTablet for tablet 9615a708166340d8bdc6e7dc90e1223b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:24 UTC) from {username='slave'} at 127.0.0.1:55040
I20260926 23:21:24.384585 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 2994
I20260926 23:21:24.419162 3425 tablet_replica.cc:330] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: stopping tablet replica
I20260926 23:21:24.419772 3425 raft_consensus.cc:2229] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:24.420463 3425 raft_consensus.cc:2258] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:24.421330 3426 tablet_replica.cc:330] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: stopping tablet replica
I20260926 23:21:24.421936 3427 tablet_replica.cc:330] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576: stopping tablet replica
I20260926 23:21:24.422349 3427 raft_consensus.cc:2229] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:21:24.423041 3427 raft_consensus.cc:2258] T 9615a708166340d8bdc6e7dc90e1223b P 327aeb357a6046b3816704d25773e576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:24.423118 3426 raft_consensus.cc:2229] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:24.423964 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.386548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.16.62.65:37133)
0926 23:21:24.386762 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.386771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.410016 (+ 23245us) server_negotiation.cc:407] Connection header received
0926 23:21:24.410568 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.410578 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.410712 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.410804 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:24.411626 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.413199 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.416326 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.417277 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.417402 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.421543 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:24.421556 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:24.421576 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:24.422413 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:24.422475 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:24.422499 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:24.422604 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.423097 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.423229 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 23:21:24.424762 3426 raft_consensus.cc:2258] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:24.424806 3116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:24.425659 3425 ts_tablet_manager.cc:1905] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:24.427266 3426 ts_tablet_manager.cc:1905] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:24.432922 3426 ts_tablet_manager.cc:1918] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:24.433082 3426 log.cc:1196] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/9615a708166340d8bdc6e7dc90e1223b
I20260926 23:21:24.433606 3426 ts_tablet_manager.cc:1939] T 9615a708166340d8bdc6e7dc90e1223b P cfd3774765a34d31a8c866b8f3828d9b: Deleting consensus metadata
I20260926 23:21:24.433723 3425 ts_tablet_manager.cc:1918] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:24.433843 3425 log.cc:1196] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/9615a708166340d8bdc6e7dc90e1223b
I20260926 23:21:24.434216 3425 ts_tablet_manager.cc:1939] T 9615a708166340d8bdc6e7dc90e1223b P d9ae385097d247a38a14e34b8c9627d3: Deleting consensus metadata
I20260926 23:21:24.435364 2921 catalog_manager.cc:4841] TS cfd3774765a34d31a8c866b8f3828d9b (127.16.62.66:42259): tablet 9615a708166340d8bdc6e7dc90e1223b (table test-table [id=c90c3a0ea3264e7183ecd38eb6691509]) successfully deleted
I20260926 23:21:24.435604 2920 catalog_manager.cc:4841] TS d9ae385097d247a38a14e34b8c9627d3 (127.16.62.67:45251): tablet 9615a708166340d8bdc6e7dc90e1223b (table test-table [id=c90c3a0ea3264e7183ecd38eb6691509]) successfully deleted
I20260926 23:21:24.526499 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 2994
I20260926 23:21:24.541332 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 3125
W20260926 23:21:24.541337 2920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.62.65:37133 (error 108)
I20260926 23:21:24.543286 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.542768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37133 (local address 127.0.0.1:55064)
0926 23:21:24.542950 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.543100 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 23:21:24.543571 2920 catalog_manager.cc:4584] TS 327aeb357a6046b3816704d25773e576 (127.16.62.65:37133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9615a708166340d8bdc6e7dc90e1223b: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37133: connect: Connection refused (error 111)
I20260926 23:21:24.548507 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.542118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.16.62.66:42259)
0926 23:21:24.542295 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.542299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.542472 (+ 173us) server_negotiation.cc:407] Connection header received
0926 23:21:24.542705 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.542710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.542823 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.542939 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:24.543433 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.544362 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.545795 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.546189 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.546281 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.547203 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:24.547210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:24.547222 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:24.547770 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:24.547814 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:24.547843 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:24.547923 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.548027 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.548105 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:21:24.549182 3247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:24.597113 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.596671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37133 (local address 127.0.0.1:55078)
0926 23:21:24.596873 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.596957 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 23:21:24.598116 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.597786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37133 (local address 127.0.0.1:55082)
0926 23:21:24.597940 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.598003 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 23:21:24.636416 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 3125
I20260926 23:21:24.653419 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 3255
I20260926 23:21:24.658268 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.657836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37133 (local address 127.0.0.1:55092)
0926 23:21:24.658032 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.658106 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 23:21:24.659243 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.658929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37133 (local address 127.0.0.1:55104)
0926 23:21:24.659073 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.659144 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 23:21:24.660389 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.654179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.16.62.67:45251)
0926 23:21:24.654356 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:24.654363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:24.654436 (+ 73us) server_negotiation.cc:407] Connection header received
0926 23:21:24.654662 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:24.654668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:24.654762 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:24.654879 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:24.655278 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.656018 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.657299 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:24.657697 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:24.657834 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:24.658945 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:24.658953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:24.658967 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:24.659579 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:24.659643 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:24.659663 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:24.659786 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:24.659880 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 23:21:24.659999 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:21:24.661024 3377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:24.739809 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.739363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37133 (local address 127.0.0.1:55108)
0926 23:21:24.739552 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.739667 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 23:21:24.740839 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:24.740448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37133 (local address 127.0.0.1:55110)
0926 23:21:24.740660 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:24.740727 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 23:21:24.750524 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 3255
I20260926 23:21:24.767165 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 2904
I20260926 23:21:24.768234 2965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:24.875317 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 2904
2026-09-26T23:21:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2669 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:24 dist-test-slave-vcv1 krb5kdc[3436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:24 dist-test-slave-vcv1 krb5kdc[3436](info): set up 2 sockets
Sep 26 23:21:24 dist-test-slave-vcv1 krb5kdc[3436](info): commencing operation
krb5kdc: starting...
W20260926 23:21:25.973323 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:26 dist-test-slave-vcv1 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:21:26.208994 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:35039
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:35039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:26.531487 3449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:26.531822 3449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:26.531981 3449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:26.539036 3449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:26.539178 3449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:26.539288 3449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:26.539383 3449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:26.539469 3449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:26.549110 3449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:35039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:35039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:26.550448 3449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:26.551715 3449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:26.563155 3454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:26.563072 3455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:26.563477 3457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:26.564286 3449 server_base.cc:1034] running on GCE node
I20260926 23:21:26.564999 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:26.566282 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:26.567507 3449 hybrid_clock.cc:648] HybridClock initialized: now 1790464886567468 us; error 46 us; skew 500 ppm
I20260926 23:21:26.567834 3449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:26 dist-test-slave-vcv1 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464886, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:26.576534 3449 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:21:26.577929 3449 webserver.cc:460] Webserver started at http://127.16.62.126:46331/ using document root <none> and password file <none>
I20260926 23:21:26.578591 3449 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:26.578748 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:26.579038 3449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:26.582033 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "3364e353a7f04155b510a0e4d35b9350"
format_stamp: "Formatted at 2026-09-26 23:21:26 on dist-test-slave-vcv1"
server_key: "fbe027183107a972b6f0ace7df9ee680"
server_key_iv: "18844f7cf27ecfb1c0ad87fd2ce60e1f"
server_key_version: "encryptionkey@0"
I20260926 23:21:26.582782 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "3364e353a7f04155b510a0e4d35b9350"
format_stamp: "Formatted at 2026-09-26 23:21:26 on dist-test-slave-vcv1"
server_key: "fbe027183107a972b6f0ace7df9ee680"
server_key_iv: "18844f7cf27ecfb1c0ad87fd2ce60e1f"
server_key_version: "encryptionkey@0"
I20260926 23:21:26.587774 3449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:21:26.591075 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:26.592391 3449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:26.592566 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "3364e353a7f04155b510a0e4d35b9350"
format_stamp: "Formatted at 2026-09-26 23:21:26 on dist-test-slave-vcv1"
server_key: "fbe027183107a972b6f0ace7df9ee680"
server_key_iv: "18844f7cf27ecfb1c0ad87fd2ce60e1f"
server_key_version: "encryptionkey@0"
I20260926 23:21:26.592758 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:26.624408 3449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:26.628684 3449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:26.628966 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:26.646929 3449 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:35039
I20260926 23:21:26.646955 3515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:35039 every 8 connection(s)
I20260926 23:21:26.648208 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:26.651796 3516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:26.652004 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 3449
I20260926 23:21:26.652240 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:26.652699 16633 external_mini_cluster.cc:1428] Setting key d1ca0d321b2d83589cda86cdf5b4ccaa
I20260926 23:21:26.665372 3516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350: Bootstrap starting.
Sep 26 23:21:26 dist-test-slave-vcv1 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:26.669075 3516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:26.670258 3516 log.cc:824] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:26.673272 3516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350: No bootstrap required, opened a new log
I20260926 23:21:26.676615 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:26.654456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.16.62.126:35039)
0926 23:21:26.655272 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:26.655283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:26.655323 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:21:26.657618 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:26.657650 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:26.658032 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:26.658352 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:26.660195 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:26.661477 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:26.663362 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:26.664017 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:26.664201 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:26.669051 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:26.669063 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:26.669077 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:26.669118 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:26.672597 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:26.673251 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:26.673265 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:26.673277 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:26.673503 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:26.673889 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:26.673895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:26.673899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:26.674411 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:26.674704 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:26.675170 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 23:21:26.675579 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":400,"threads_started":1}
I20260926 23:21:26.681041 3516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3364e353a7f04155b510a0e4d35b9350" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35039 } }
I20260926 23:21:26.681402 3516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:26.681509 3516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3364e353a7f04155b510a0e4d35b9350, State: Initialized, Role: FOLLOWER
I20260926 23:21:26.682054 3516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3364e353a7f04155b510a0e4d35b9350" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35039 } }
I20260926 23:21:26.682302 3516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:26.682444 3516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:26.682621 3516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:26.684768 3516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3364e353a7f04155b510a0e4d35b9350" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35039 } }
I20260926 23:21:26.685232 3516 leader_election.cc:302] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3364e353a7f04155b510a0e4d35b9350; no voters:
I20260926 23:21:26.685787 3516 leader_election.cc:288] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:26.685910 3521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:26.686545 3521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [term 1 LEADER]: Becoming Leader. State: Replica: 3364e353a7f04155b510a0e4d35b9350, State: Running, Role: LEADER
I20260926 23:21:26.687101 3521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3364e353a7f04155b510a0e4d35b9350" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35039 } }
I20260926 23:21:26.687736 3516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:26.689597 3522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3364e353a7f04155b510a0e4d35b9350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3364e353a7f04155b510a0e4d35b9350" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35039 } } }
I20260926 23:21:26.690021 3522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:26.690497 3523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3364e353a7f04155b510a0e4d35b9350. Latest consensus state: current_term: 1 leader_uuid: "3364e353a7f04155b510a0e4d35b9350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3364e353a7f04155b510a0e4d35b9350" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 35039 } } }
I20260926 23:21:26.692402 3523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:26.693419 3530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:26.698268 3530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:26.706187 3530 catalog_manager.cc:1347] Generated new cluster ID: 5dc9a80bf836439483149b45c17a0939
I20260926 23:21:26.706429 3530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:26.719853 3530 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:26.721086 3530 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:26.726209 3530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350: Generated new TSK 0
I20260926 23:21:26.726933 3530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.62.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
Entry for principal kudu/127.16.62.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab.
I20260926 23:21:26.792855 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35039
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:27.115804 3542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:27.116103 3542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:27.116381 3542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:27.123281 3542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:27.123383 3542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:27.123567 3542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:21:27.133038 3542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.65.keytab
--principal=kudu/127.16.62.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:27.134200 3542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:27.135489 3542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:27.148154 3547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:27.148272 3548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:27.148240 3550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:27.149366 3542 server_base.cc:1034] running on GCE node
I20260926 23:21:27.149948 3542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:27.150804 3542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:27.152029 3542 hybrid_clock.cc:648] HybridClock initialized: now 1790464887151989 us; error 47 us; skew 500 ppm
I20260926 23:21:27.152352 3542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:27 dist-test-slave-vcv1 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:27.155364 3542 init.cc:377] Logged in from keytab as kudu/127.16.62.65@KRBTEST.COM (short username kudu)
I20260926 23:21:27.156791 3542 webserver.cc:460] Webserver started at http://127.16.62.65:33829/ using document root <none> and password file <none>
I20260926 23:21:27.157440 3542 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:27.157577 3542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:27.157878 3542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:27.160830 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "985832e44b294445ade05384617b614b"
format_stamp: "Formatted at 2026-09-26 23:21:27 on dist-test-slave-vcv1"
server_key: "eb8518f84e47ab7c6e50cbc28514acae"
server_key_iv: "df74877b502251ae5b33baf729008193"
server_key_version: "encryptionkey@0"
I20260926 23:21:27.161573 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "985832e44b294445ade05384617b614b"
format_stamp: "Formatted at 2026-09-26 23:21:27 on dist-test-slave-vcv1"
server_key: "eb8518f84e47ab7c6e50cbc28514acae"
server_key_iv: "df74877b502251ae5b33baf729008193"
server_key_version: "encryptionkey@0"
I20260926 23:21:27.166421 3542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:21:27.169797 3557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:27.171010 3542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:21:27.171180 3542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "985832e44b294445ade05384617b614b"
format_stamp: "Formatted at 2026-09-26 23:21:27 on dist-test-slave-vcv1"
server_key: "eb8518f84e47ab7c6e50cbc28514acae"
server_key_iv: "df74877b502251ae5b33baf729008193"
server_key_version: "encryptionkey@0"
I20260926 23:21:27.171368 3542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:27.191316 3542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:27.195298 3542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:27.195577 3542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:27.196607 3542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:27.197911 3542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:27.198011 3542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:27.198160 3542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:27.198231 3542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:27.230400 3542 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:37201
I20260926 23:21:27.230445 3669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:37201 every 8 connection(s)
I20260926 23:21:27.231597 3542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:21:27.237643 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 3542
I20260926 23:21:27.237830 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:21:27.238256 16633 external_mini_cluster.cc:1428] Setting key c1af32d2646d8156447ae1e8af3e8684
Sep 26 23:21:27 dist-test-slave-vcv1 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.65@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:27.251912 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:27.234005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:53011 (local address 127.16.62.126:35039)
0926 23:21:27.234431 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:27.234435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:27.235265 (+ 830us) server_negotiation.cc:407] Connection header received
0926 23:21:27.237176 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:27.237182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:27.237269 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:27.237386 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:27.238949 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:27.239948 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:27.241498 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:27.241903 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:27.242036 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:27.245819 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:27.245824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:27.245827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:27.245859 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:27.248817 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:27.249507 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:27.249516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:27.249522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:27.249668 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:27.250041 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:27.250048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:27.250054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:27.250418 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:27.250575 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:27.251198 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 23:21:27.251452 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":231,"threads_started":1}
I20260926 23:21:27.252439 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:27.234449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35039 (local address 127.16.62.65:53011)
0926 23:21:27.235110 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:27.235153 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:27.236907 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:27.237620 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:27.237628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:27.238213 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:27.238782 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:27.238799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:27.240113 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:27.240117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:27.241355 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:27.241366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:27.242121 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:27.242130 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:27.242355 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:27.243000 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:21:27.243024 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:27.245600 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:27.248956 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:27.248962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:27.248972 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:27.249376 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:27.249768 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:27.249772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:27.249774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:27.249932 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:27.250530 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:27.250535 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:27.250822 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 23:21:27.251025 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 23:21:27.251371 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":255,"threads_started":1}
I20260926 23:21:27.253906 3670 heartbeater.cc:347] Connected to a master server at 127.16.62.126:35039
I20260926 23:21:27.254204 3670 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:27.254879 3670 heartbeater.cc:511] Master 127.16.62.126:35039 requested a full tablet report, sending...
I20260926 23:21:27.256888 3481 ts_manager.cc:194] Registered new tserver with Master: 985832e44b294445ade05384617b614b (127.16.62.65:37201)
I20260926 23:21:27.258299 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.65@KRBTEST.COM'} at 127.16.62.65:53011
WARNING: no policy specified for kudu/127.16.62.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
Entry for principal kudu/127.16.62.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab.
I20260926 23:21:27.294011 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35039
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:27.608201 3676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:27.608502 3676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:27.608673 3676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:27.615599 3676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:27.615759 3676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:27.615928 3676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:21:27.624955 3676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.66.keytab
--principal=kudu/127.16.62.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:27.626096 3676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:27.627295 3676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:27.639731 3684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:27.639791 3681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:27.639791 3682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:27.640537 3676 server_base.cc:1034] running on GCE node
I20260926 23:21:27.641074 3676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:27.641855 3676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:27.643043 3676 hybrid_clock.cc:648] HybridClock initialized: now 1790464887642999 us; error 33 us; skew 500 ppm
I20260926 23:21:27.643339 3676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:27 dist-test-slave-vcv1 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:27.646570 3676 init.cc:377] Logged in from keytab as kudu/127.16.62.66@KRBTEST.COM (short username kudu)
I20260926 23:21:27.648177 3676 webserver.cc:460] Webserver started at http://127.16.62.66:42797/ using document root <none> and password file <none>
I20260926 23:21:27.648954 3676 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:27.649078 3676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:27.649379 3676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:27.652930 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "ed0e4748fbc64023be08e625f893fa09"
format_stamp: "Formatted at 2026-09-26 23:21:27 on dist-test-slave-vcv1"
server_key: "c23915500c1223e14924cafe17bc5cb6"
server_key_iv: "5321ab89ae0fae5600e6947eeab540cd"
server_key_version: "encryptionkey@0"
I20260926 23:21:27.653833 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "ed0e4748fbc64023be08e625f893fa09"
format_stamp: "Formatted at 2026-09-26 23:21:27 on dist-test-slave-vcv1"
server_key: "c23915500c1223e14924cafe17bc5cb6"
server_key_iv: "5321ab89ae0fae5600e6947eeab540cd"
server_key_version: "encryptionkey@0"
I20260926 23:21:27.659751 3676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:21:27.663800 3691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:27.665283 3676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:27.665414 3676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "ed0e4748fbc64023be08e625f893fa09"
format_stamp: "Formatted at 2026-09-26 23:21:27 on dist-test-slave-vcv1"
server_key: "c23915500c1223e14924cafe17bc5cb6"
server_key_iv: "5321ab89ae0fae5600e6947eeab540cd"
server_key_version: "encryptionkey@0"
I20260926 23:21:27.665616 3676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:27.683562 3676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:27.687482 3676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:27.687768 3676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:27.688764 3676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:27.690007 3676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:27.690078 3676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:27.690167 3676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:27.690210 3676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:27.722186 3676 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:43353
I20260926 23:21:27.722266 3803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:43353 every 8 connection(s)
I20260926 23:21:27.723436 3676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:21:27.726569 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 3676
I20260926 23:21:27.726785 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:21:27.727367 16633 external_mini_cluster.cc:1428] Setting key e8133f7a263809cb630ee0d43d96769c
Sep 26 23:21:27 dist-test-slave-vcv1 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.66@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:27.745638 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:27.725866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:45267 (local address 127.16.62.126:35039)
0926 23:21:27.726038 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:27.726043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:27.727362 (+ 1319us) server_negotiation.cc:407] Connection header received
0926 23:21:27.729538 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:27.729544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:27.729625 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:27.729736 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:27.731149 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:27.731866 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:27.733446 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:27.733867 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:27.733994 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:27.737769 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:27.737774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:27.737777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:27.737807 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:27.741383 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:27.742346 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:27.742355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:27.742362 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:27.742543 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:27.743179 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:27.743187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:27.743194 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:27.743626 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:27.743777 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:27.744884 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0926 23:21:27.745169 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:21:27.746807 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:27.726426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35039 (local address 127.16.62.66:45267)
0926 23:21:27.727219 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:27.727254 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:27.729291 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:27.729961 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:27.729969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:27.730447 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:27.730977 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:27.730994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:27.732013 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:27.732018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:27.733298 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:27.733310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:27.734091 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:27.734098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:27.734348 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:27.735053 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:21:27.735079 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:27.737538 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:27.741538 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:27.741549 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:27.741566 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:27.742191 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:27.742712 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:27.742719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:27.742725 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:27.743016 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:27.743882 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:27.743890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:27.744379 (+ 489us) client_negotiation.cc:769] Sending connection context
0926 23:21:27.744758 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 23:21:27.745323 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":360,"threads_started":1}
I20260926 23:21:27.748641 3804 heartbeater.cc:347] Connected to a master server at 127.16.62.126:35039
I20260926 23:21:27.748982 3804 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:27.749720 3804 heartbeater.cc:511] Master 127.16.62.126:35039 requested a full tablet report, sending...
I20260926 23:21:27.751390 3481 ts_manager.cc:194] Registered new tserver with Master: ed0e4748fbc64023be08e625f893fa09 (127.16.62.66:43353)
I20260926 23:21:27.752727 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.66@KRBTEST.COM'} at 127.16.62.66:45267
WARNING: no policy specified for kudu/127.16.62.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
Entry for principal kudu/127.16.62.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab.
I20260926 23:21:27.785008 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35039
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:28.099855 3809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:28.100122 3809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:28.100268 3809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:28.107316 3809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:28.107405 3809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:28.107635 3809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:21:28.116953 3809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.67.keytab
--principal=kudu/127.16.62.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:35039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:28.118108 3809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:28.119361 3809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:28.132035 3817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:28.132115 3815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:28.132593 3814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:28.133106 3809 server_base.cc:1034] running on GCE node
I20260926 23:21:28.133759 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:28.134645 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:28.135844 3809 hybrid_clock.cc:648] HybridClock initialized: now 1790464888135800 us; error 33 us; skew 500 ppm
I20260926 23:21:28.136255 3809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:28 dist-test-slave-vcv1 krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:28.139484 3809 init.cc:377] Logged in from keytab as kudu/127.16.62.67@KRBTEST.COM (short username kudu)
I20260926 23:21:28.140949 3809 webserver.cc:460] Webserver started at http://127.16.62.67:36217/ using document root <none> and password file <none>
I20260926 23:21:28.141584 3809 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:28.141732 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:28.142032 3809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:28.144882 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "bf24d321a0dc44a09e2997e83b5c3f9e"
format_stamp: "Formatted at 2026-09-26 23:21:28 on dist-test-slave-vcv1"
server_key: "012d12ebb24b9fa7a2d76a6ec624de43"
server_key_iv: "e58edb7583d6f26801cc27ea3322c116"
server_key_version: "encryptionkey@0"
I20260926 23:21:28.145618 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "bf24d321a0dc44a09e2997e83b5c3f9e"
format_stamp: "Formatted at 2026-09-26 23:21:28 on dist-test-slave-vcv1"
server_key: "012d12ebb24b9fa7a2d76a6ec624de43"
server_key_iv: "e58edb7583d6f26801cc27ea3322c116"
server_key_version: "encryptionkey@0"
I20260926 23:21:28.150362 3809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 23:21:28.153530 3824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:28.154721 3809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 23:21:28.154882 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "bf24d321a0dc44a09e2997e83b5c3f9e"
format_stamp: "Formatted at 2026-09-26 23:21:28 on dist-test-slave-vcv1"
server_key: "012d12ebb24b9fa7a2d76a6ec624de43"
server_key_iv: "e58edb7583d6f26801cc27ea3322c116"
server_key_version: "encryptionkey@0"
I20260926 23:21:28.155073 3809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:28.171458 3809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:28.175308 3809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:28.175570 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:28.176496 3809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:28.177692 3809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:28.177775 3809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:28.177944 3809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:28.177992 3809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:28.209143 3809 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:42979
I20260926 23:21:28.209173 3936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:42979 every 8 connection(s)
I20260926 23:21:28.210356 3809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:21:28.215981 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 3809
I20260926 23:21:28.216215 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:21:28.216777 16633 external_mini_cluster.cc:1428] Setting key 2b0738c19861b58d88fd4044ec0ef469
Sep 26 23:21:28 dist-test-slave-vcv1 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.67@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:28.229326 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.212890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:55169 (local address 127.16.62.126:35039)
0926 23:21:28.213062 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.213066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.214092 (+ 1026us) server_negotiation.cc:407] Connection header received
0926 23:21:28.216140 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.216165 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.216298 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.216397 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:28.218056 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.218811 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.220278 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.220665 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.220802 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.223944 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:28.223950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:28.223953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:28.223984 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:28.226420 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.227095 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.227103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.227108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.227264 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.227662 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.227669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.227672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.227946 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:28.228102 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.228758 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.228982 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:21:28.230477 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.213267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:35039 (local address 127.16.62.67:55169)
0926 23:21:28.213931 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.213967 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:28.215801 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:28.216655 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:28.216666 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:28.217348 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:28.217878 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.217894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.218953 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.218964 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.220097 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.220108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.220885 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.220893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.221139 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.221731 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:21:28.221756 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:28.223751 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:28.226555 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:28.226563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:28.226577 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:28.226976 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:28.227381 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:21:28.227386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:21:28.227390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:21:28.227550 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:21:28.228073 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:28.228080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:28.228409 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 23:21:28.228678 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 23:21:28.229145 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":270,"threads_started":1}
I20260926 23:21:28.231958 3937 heartbeater.cc:347] Connected to a master server at 127.16.62.126:35039
I20260926 23:21:28.232286 3937 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:28.233103 3937 heartbeater.cc:511] Master 127.16.62.126:35039 requested a full tablet report, sending...
I20260926 23:21:28.234678 3480 ts_manager.cc:194] Registered new tserver with Master: bf24d321a0dc44a09e2997e83b5c3f9e (127.16.62.67:42979)
I20260926 23:21:28.235507 3480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.62.67@KRBTEST.COM'} at 127.16.62.67:55169
I20260926 23:21:28.243520 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:21:28.260390 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.247526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.16.62.126:35039)
0926 23:21:28.247685 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.247689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.248225 (+ 536us) server_negotiation.cc:407] Connection header received
0926 23:21:28.248384 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.248388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.248458 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.248534 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:28.250006 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.250947 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.252221 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.252619 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.252712 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.254920 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:28.254925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:28.254928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:28.254958 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:28.257717 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.258295 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.258299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.258301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.258441 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.258949 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.258952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.258954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.259331 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:28.259508 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.259888 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.260053 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:21:28.261929 3670 heartbeater.cc:503] Master 127.16.62.126:35039 was elected leader, sending a full tablet report...
I20260926 23:21:28.264199 3480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:21:28.270699 3480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:21:28.291917 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.282479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.16.62.67:42979)
0926 23:21:28.283538 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.283547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.283578 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:21:28.283690 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.283700 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.283957 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.284135 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:28.284892 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.286523 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.289216 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.290620 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.290752 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.290980 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.291408 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.291566 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":746,"threads_started":1}
I20260926 23:21:28.291982 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.282288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:42979 (local address 127.0.0.1:42238)
0926 23:21:28.283107 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.283157 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:28.283329 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:28.284302 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:28.284311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:28.284338 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:28.284687 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.284700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.286631 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.286638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.289027 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.289038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.290817 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.290823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.291112 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.291179 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 23:21:28.291352 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 23:21:28.291524 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":79,"thread_start_us":475,"threads_started":1}
I20260926 23:21:28.292725 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.283778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.16.62.66:43353)
0926 23:21:28.284760 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.284769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.284796 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:21:28.284890 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.284896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.285671 (+ 775us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.285834 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:28.286534 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.287830 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.290113 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.291235 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.291378 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.291535 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.292246 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.292416 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":693,"threads_started":1}
I20260926 23:21:28.293345 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.283458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:43353 (local address 127.0.0.1:51374)
0926 23:21:28.283981 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.284001 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:28.284114 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:28.286040 (+ 1926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:28.286045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:28.286067 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:28.286374 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.286384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.287945 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.287953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.289972 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.289983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.291693 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.291700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.291872 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.291904 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:21:28.292008 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 23:21:28.292899 (+ 891us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":1,"thread_start_us":252,"threads_started":1}
I20260926 23:21:28.294840 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.284067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.0.0.1:59654)
0926 23:21:28.285619 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.285635 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:28.285814 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:28.286546 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:28.286550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:28.286572 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:28.287004 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.287016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.288969 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.288978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.291120 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.291138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.294037 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.294043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.294242 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.294274 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:21:28.294366 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:21:28.294482 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1425,"mutex_wait_us":57,"thread_start_us":1299,"threads_started":1}
I20260926 23:21:28.294889 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.284532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.16.62.65:37201)
0926 23:21:28.285168 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.285179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.285726 (+ 547us) server_negotiation.cc:407] Connection header received
0926 23:21:28.285993 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.286000 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.286209 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.286318 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:28.287234 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.288798 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.291287 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.293344 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.293661 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.293903 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.294416 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.294590 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":275,"threads_started":1}
I20260926 23:21:28.296088 3872 tablet_service.cc:1467] Processing CreateTablet for tablet 773d18bab17542709c8e57570bc41c65 (DEFAULT_TABLE table=test-table [id=b3683af9d8494355a973e12303c9438b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:28.296722 3739 tablet_service.cc:1467] Processing CreateTablet for tablet 773d18bab17542709c8e57570bc41c65 (DEFAULT_TABLE table=test-table [id=b3683af9d8494355a973e12303c9438b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:28.297699 3872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773d18bab17542709c8e57570bc41c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:28.298318 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773d18bab17542709c8e57570bc41c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:28.298470 3605 tablet_service.cc:1467] Processing CreateTablet for tablet 773d18bab17542709c8e57570bc41c65 (DEFAULT_TABLE table=test-table [id=b3683af9d8494355a973e12303c9438b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:28.299733 3605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 773d18bab17542709c8e57570bc41c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:28.308113 3955 tablet_bootstrap.cc:492] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: Bootstrap starting.
I20260926 23:21:28.312103 3956 tablet_bootstrap.cc:492] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: Bootstrap starting.
I20260926 23:21:28.312458 3955 tablet_bootstrap.cc:654] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:28.313467 3957 tablet_bootstrap.cc:492] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b: Bootstrap starting.
I20260926 23:21:28.314180 3955 log.cc:824] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:28.315683 3956 tablet_bootstrap.cc:654] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:28.316398 3957 tablet_bootstrap.cc:654] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:28.316953 3956 log.cc:824] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:28.317564 3957 log.cc:824] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:28.319867 3957 tablet_bootstrap.cc:492] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b: No bootstrap required, opened a new log
I20260926 23:21:28.319993 3955 tablet_bootstrap.cc:492] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: No bootstrap required, opened a new log
I20260926 23:21:28.320186 3957 ts_tablet_manager.cc:1397] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 23:21:28.320331 3955 ts_tablet_manager.cc:1397] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260926 23:21:28.326737 3956 tablet_bootstrap.cc:492] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: No bootstrap required, opened a new log
I20260926 23:21:28.326985 3956 ts_tablet_manager.cc:1397] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260926 23:21:28.328486 3957 raft_consensus.cc:348] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } }
I20260926 23:21:28.328842 3957 raft_consensus.cc:374] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:28.328941 3957 raft_consensus.cc:729] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985832e44b294445ade05384617b614b, State: Initialized, Role: FOLLOWER
I20260926 23:21:28.329535 3957 consensus_queue.cc:260] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } }
I20260926 23:21:28.331184 3957 ts_tablet_manager.cc:1428] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b: Time spent starting tablet: real 0.011s user 0.001s sys 0.008s
I20260926 23:21:28.331403 3955 raft_consensus.cc:348] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } }
I20260926 23:21:28.331806 3955 raft_consensus.cc:374] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:28.331890 3955 raft_consensus.cc:729] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf24d321a0dc44a09e2997e83b5c3f9e, State: Initialized, Role: FOLLOWER
I20260926 23:21:28.332456 3955 consensus_queue.cc:260] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } }
I20260926 23:21:28.333524 3937 heartbeater.cc:503] Master 127.16.62.126:35039 was elected leader, sending a full tablet report...
I20260926 23:21:28.334050 3955 ts_tablet_manager.cc:1428] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260926 23:21:28.337203 3956 raft_consensus.cc:348] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } }
I20260926 23:21:28.337602 3956 raft_consensus.cc:374] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:28.337694 3956 raft_consensus.cc:729] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0e4748fbc64023be08e625f893fa09, State: Initialized, Role: FOLLOWER
I20260926 23:21:28.338231 3956 consensus_queue.cc:260] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } }
I20260926 23:21:28.339148 3804 heartbeater.cc:503] Master 127.16.62.126:35039 was elected leader, sending a full tablet report...
I20260926 23:21:28.339753 3956 ts_tablet_manager.cc:1428] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260926 23:21:28.461959 3938 tablet.cc:2369] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:21:28.475425 3805 tablet.cc:2369] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:21:28.484305 3671 tablet.cc:2369] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:21:28.516319 3962 raft_consensus.cc:482] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:21:28.516647 3962 raft_consensus.cc:504] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } }
I20260926 23:21:28.518574 3962 leader_election.cc:288] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed0e4748fbc64023be08e625f893fa09 (127.16.62.66:43353), 985832e44b294445ade05384617b614b (127.16.62.65:37201)
I20260926 23:21:28.524973 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.518796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:43353 (local address 127.16.62.67:51043)
0926 23:21:28.519044 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.519066 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:28.519207 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:28.519785 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:28.519790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:28.519846 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:28.520176 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.520188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.521461 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.521467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.523117 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.523129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.524223 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.524228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.524418 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.524442 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 23:21:28.524550 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 23:21:28.524671 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"spinlock_wait_cycles":6912}
I20260926 23:21:28.524968 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.519028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:51043 (local address 127.16.62.66:43353)
0926 23:21:28.519261 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.519270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.519291 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:21:28.519371 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.519377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.519468 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.519590 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:28.520345 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.521319 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.523274 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.524059 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.524259 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.524337 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.524586 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.524712 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 23:21:28.525552 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.519423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:37305 (local address 127.16.62.65:37201)
0926 23:21:28.519628 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.519656 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.519970 (+ 314us) server_negotiation.cc:407] Connection header received
0926 23:21:28.520081 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.520089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.520195 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.520292 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:28.520870 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.521877 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.523696 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.524582 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.524763 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.524833 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.525171 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.525296 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:21:28.526196 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.519173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.16.62.67:37305)
0926 23:21:28.519840 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.519855 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:28.519972 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:28.520335 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:28.520339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:28.520364 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:28.520720 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.520732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.522039 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.522044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.523540 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:28.523551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.524782 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.524788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:28.524957 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.524984 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 23:21:28.525740 (+ 756us) client_negotiation.cc:240] Negotiation successful
0926 23:21:28.525867 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":304,"threads_started":1}
I20260926 23:21:28.526051 3759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773d18bab17542709c8e57570bc41c65" candidate_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0e4748fbc64023be08e625f893fa09" is_pre_election: true
I20260926 23:21:28.526628 3759 raft_consensus.cc:2454] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf24d321a0dc44a09e2997e83b5c3f9e in term 0.
I20260926 23:21:28.527269 3625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773d18bab17542709c8e57570bc41c65" candidate_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985832e44b294445ade05384617b614b" is_pre_election: true
I20260926 23:21:28.527518 3828 leader_election.cc:302] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf24d321a0dc44a09e2997e83b5c3f9e, ed0e4748fbc64023be08e625f893fa09; no voters:
I20260926 23:21:28.527784 3625 raft_consensus.cc:2454] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf24d321a0dc44a09e2997e83b5c3f9e in term 0.
I20260926 23:21:28.527977 3962 raft_consensus.cc:2781] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:21:28.528141 3962 raft_consensus.cc:482] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:21:28.528301 3962 raft_consensus.cc:3037] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:28.530496 3962 raft_consensus.cc:504] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } }
I20260926 23:21:28.531081 3962 leader_election.cc:288] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [CANDIDATE]: Term 1 election: Requested vote from peers ed0e4748fbc64023be08e625f893fa09 (127.16.62.66:43353), 985832e44b294445ade05384617b614b (127.16.62.65:37201)
I20260926 23:21:28.531486 3759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773d18bab17542709c8e57570bc41c65" candidate_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0e4748fbc64023be08e625f893fa09"
I20260926 23:21:28.531561 3625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773d18bab17542709c8e57570bc41c65" candidate_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985832e44b294445ade05384617b614b"
I20260926 23:21:28.531683 3759 raft_consensus.cc:3037] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:28.531790 3625 raft_consensus.cc:3037] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:28.534004 3759 raft_consensus.cc:2454] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf24d321a0dc44a09e2997e83b5c3f9e in term 1.
I20260926 23:21:28.534195 3625 raft_consensus.cc:2454] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf24d321a0dc44a09e2997e83b5c3f9e in term 1.
I20260926 23:21:28.534412 3828 leader_election.cc:302] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf24d321a0dc44a09e2997e83b5c3f9e, ed0e4748fbc64023be08e625f893fa09; no voters:
I20260926 23:21:28.534685 3962 raft_consensus.cc:2781] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:28.535125 3962 raft_consensus.cc:686] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 1 LEADER]: Becoming Leader. State: Replica: bf24d321a0dc44a09e2997e83b5c3f9e, State: Running, Role: LEADER
I20260926 23:21:28.535590 3962 consensus_queue.cc:237] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } }
I20260926 23:21:28.540062 3481 catalog_manager.cc:5495] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf24d321a0dc44a09e2997e83b5c3f9e (127.16.62.67). New cstate: current_term: 1 leader_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf24d321a0dc44a09e2997e83b5c3f9e" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 } health_report { overall_health: UNKNOWN } } }
I20260926 23:21:28.590364 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.585389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.16.62.67:42979)
0926 23:21:28.585591 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.585595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.585612 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:21:28.585775 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.585779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.585853 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.585970 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:21:28.586321 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.587037 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.588425 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.588811 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.588902 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.589351 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:21:28.589811 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:21:28.589921 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.590000 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.590060 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:21:28.598215 3625 raft_consensus.cc:1261] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [term 1 FOLLOWER]: Refusing update from remote peer bf24d321a0dc44a09e2997e83b5c3f9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:28.598232 3759 raft_consensus.cc:1261] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [term 1 FOLLOWER]: Refusing update from remote peer bf24d321a0dc44a09e2997e83b5c3f9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:28.599105 3965 consensus_queue.cc:1035] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "985832e44b294445ade05384617b614b" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 37201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:28.599442 3962 consensus_queue.cc:1035] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0e4748fbc64023be08e625f893fa09" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 43353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:28.624691 3481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:28.624987 3481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:28.628437 3481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3364e353a7f04155b510a0e4d35b9350: Sending DeleteTablet for 3 replicas of tablet 773d18bab17542709c8e57570bc41c65
I20260926 23:21:28.629473 3872 tablet_service.cc:1514] Processing DeleteTablet for tablet 773d18bab17542709c8e57570bc41c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:28 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:42238
I20260926 23:21:28.629866 3739 tablet_service.cc:1514] Processing DeleteTablet for tablet 773d18bab17542709c8e57570bc41c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:28 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:51374
I20260926 23:21:28.629908 3605 tablet_service.cc:1514] Processing DeleteTablet for tablet 773d18bab17542709c8e57570bc41c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:28 UTC) from {username='kudu', principal='kudu/127.16.62.126@KRBTEST.COM'} at 127.0.0.1:59654
I20260926 23:21:28.633334 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 3542
I20260926 23:21:28.644778 3979 tablet_replica.cc:330] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: stopping tablet replica
I20260926 23:21:28.645320 3979 raft_consensus.cc:2229] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:28.645889 3979 raft_consensus.cc:2258] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:28.648456 3979 ts_tablet_manager.cc:1905] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:28.651446 3978 tablet_replica.cc:330] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: stopping tablet replica
I20260926 23:21:28.651826 3978 raft_consensus.cc:2229] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:21:28.652473 3978 raft_consensus.cc:2258] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:28.655733 3979 ts_tablet_manager.cc:1918] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:28.655944 3979 log.cc:1196] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/773d18bab17542709c8e57570bc41c65
I20260926 23:21:28.656631 3979 ts_tablet_manager.cc:1939] T 773d18bab17542709c8e57570bc41c65 P ed0e4748fbc64023be08e625f893fa09: Deleting consensus metadata
Sep 26 23:21:28 dist-test-slave-vcv1 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.65@KRBTEST.COM
I20260926 23:21:28.658622 3468 catalog_manager.cc:4841] TS ed0e4748fbc64023be08e625f893fa09 (127.16.62.66:43353): tablet 773d18bab17542709c8e57570bc41c65 (table test-table [id=b3683af9d8494355a973e12303c9438b]) successfully deleted
I20260926 23:21:28.659544 3978 ts_tablet_manager.cc:1905] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:28.660657 3980 tablet_replica.cc:330] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b: stopping tablet replica
I20260926 23:21:28.661046 3980 raft_consensus.cc:2229] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:28.661545 3980 raft_consensus.cc:2258] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:28.663571 3980 ts_tablet_manager.cc:1905] T 773d18bab17542709c8e57570bc41c65 P 985832e44b294445ade05384617b614b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:28.674011 3978 ts_tablet_manager.cc:1918] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:28.674158 3978 log.cc:1196] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/773d18bab17542709c8e57570bc41c65
I20260926 23:21:28.674729 3978 ts_tablet_manager.cc:1939] T 773d18bab17542709c8e57570bc41c65 P bf24d321a0dc44a09e2997e83b5c3f9e: Deleting consensus metadata
I20260926 23:21:28.676479 3468 catalog_manager.cc:4841] TS bf24d321a0dc44a09e2997e83b5c3f9e (127.16.62.67:42979): tablet 773d18bab17542709c8e57570bc41c65 (table test-table [id=b3683af9d8494355a973e12303c9438b]) successfully deleted
I20260926 23:21:28.677264 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.634837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.16.62.65:37201)
0926 23:21:28.635006 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.635013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.639381 (+ 4368us) server_negotiation.cc:407] Connection header received
0926 23:21:28.640316 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.640324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.640406 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.640507 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:28.646774 (+ 6267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.647944 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.652909 (+ 4965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.653591 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.653691 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.665325 (+ 11634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:28.665340 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:28.665356 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:28.665395 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:28.669096 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.669893 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.669903 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.669909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.670094 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.670564 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.670571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.670575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.671077 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:28.671385 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.671645 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.671864 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:21:28.677932 3665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:28.773309 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 3542
I20260926 23:21:28.791638 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 3676
W20260926 23:21:28.791635 3468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.62.65:37201 (error 108)
I20260926 23:21:28.793776 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.793211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.0.0.1:59664)
0926 23:21:28.793418 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.793547 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260926 23:21:28.794095 3468 catalog_manager.cc:4584] TS 985832e44b294445ade05384617b614b (127.16.62.65:37201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 773d18bab17542709c8e57570bc41c65: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37201: connect: Connection refused (error 111)
Sep 26 23:21:28 dist-test-slave-vcv1 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.66@KRBTEST.COM
I20260926 23:21:28.807235 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.792448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.16.62.66:43353)
0926 23:21:28.792732 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.792740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.792764 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:21:28.792951 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.792955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.793028 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.793140 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:28.794667 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.795428 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.796925 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.797315 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.797451 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.801571 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:28.801585 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:28.801598 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:28.801632 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:28.804435 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.805107 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.805113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.805116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.805249 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.805624 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.805631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.805634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.805979 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:28.806327 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.806686 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.806899 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260926 23:21:28.807816 3799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:28.847924 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.847385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.0.0.1:59666)
0926 23:21:28.847632 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.847753 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 23:21:28.849133 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.848689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.0.0.1:59670)
0926 23:21:28.848850 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.848975 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 23:21:28.898267 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 3676
I20260926 23:21:28.909983 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.909388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.0.0.1:59674)
0926 23:21:28.909640 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.909761 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 23:21:28.911304 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.910822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.0.0.1:59678)
0926 23:21:28.911032 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.911132 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 23:21:28.916622 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 3809
Sep 26 23:21:28 dist-test-slave-vcv1 krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.67@KRBTEST.COM
I20260926 23:21:28.932283 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.917344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.16.62.67:42979)
0926 23:21:28.917537 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:28.917542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:28.917562 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:21:28.917753 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:28.917765 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:28.917825 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:28.917927 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:28.919224 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.920028 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.921323 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:28.921763 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:28.921904 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:28.926256 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:28.926266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:28.926281 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:28.926314 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:28.929440 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.930105 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.930112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.930117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.930276 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:28.930666 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:28.930672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:28.930675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:28.931016 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:28.931347 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:28.931614 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 23:21:28.931830 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:21:28.933048 3932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:28.991762 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.991191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.0.0.1:59694)
0926 23:21:28.991429 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.991553 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 23:21:28.992928 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:28.992492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.0.0.1:59702)
0926 23:21:28.992707 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:28.992792 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 23:21:29.054731 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 3809
I20260926 23:21:29.073839 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 3449
I20260926 23:21:29.074918 3511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:29.185307 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:29.184677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:37201 (local address 127.0.0.1:59704)
0926 23:21:29.184888 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:29.185005 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:37201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 23:21:29.185775 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 3449
2026-09-26T23:21:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4311 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T23:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:29Z Disabled control of system clock
I20260926 23:21:29.261327 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44789
--webserver_interface=127.16.62.126
--webserver_port=0
--builtin_ntp_servers=127.16.62.84:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:44789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:21:29.580101 3990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:29.580437 3990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:29.580593 3990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:29.587539 3990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:29.587637 3990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:29.587731 3990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:29.587828 3990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:29.587908 3990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:29.596962 3990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:44789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:44789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:29.598142 3990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:29.599876 3990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:29.611032 3998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:29.611130 3995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:29.611105 3996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:29.611852 3990 server_base.cc:1034] running on GCE node
I20260926 23:21:29.612557 3990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:29.613740 3990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:29.614928 3990 hybrid_clock.cc:648] HybridClock initialized: now 1790464889614890 us; error 30 us; skew 500 ppm
I20260926 23:21:29.615212 3990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:29.617491 3990 webserver.cc:460] Webserver started at http://127.16.62.126:37821/ using document root <none> and password file <none>
I20260926 23:21:29.618074 3990 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:29.618165 3990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:29.618413 3990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:29.621114 3990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "44e20a28b4d24105b63ccef77dff84c1"
format_stamp: "Formatted at 2026-09-26 23:21:29 on dist-test-slave-vcv1"
server_key: "6f3eb17ea4a7a984ebff6b614f43afd0"
server_key_iv: "d9504e4f3d2558d6b663a11d7b4b0cd3"
server_key_version: "encryptionkey@0"
I20260926 23:21:29.621836 3990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "44e20a28b4d24105b63ccef77dff84c1"
format_stamp: "Formatted at 2026-09-26 23:21:29 on dist-test-slave-vcv1"
server_key: "6f3eb17ea4a7a984ebff6b614f43afd0"
server_key_iv: "d9504e4f3d2558d6b663a11d7b4b0cd3"
server_key_version: "encryptionkey@0"
I20260926 23:21:29.626856 3990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:21:29.630817 4004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:29.632447 3990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:29.632609 3990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "44e20a28b4d24105b63ccef77dff84c1"
format_stamp: "Formatted at 2026-09-26 23:21:29 on dist-test-slave-vcv1"
server_key: "6f3eb17ea4a7a984ebff6b614f43afd0"
server_key_iv: "d9504e4f3d2558d6b663a11d7b4b0cd3"
server_key_version: "encryptionkey@0"
I20260926 23:21:29.632819 3990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:29.645364 3990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:29.646124 3990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:29.646416 3990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:29.664415 3990 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:44789
I20260926 23:21:29.664453 4055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:44789 every 8 connection(s)
I20260926 23:21:29.665751 3990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:29.669426 4056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:29.673348 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 3990
I20260926 23:21:29.673550 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:29.674077 16633 external_mini_cluster.cc:1428] Setting key 45149b548e8d83aec1d5414b656985fa
I20260926 23:21:29.681763 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1: Bootstrap starting.
I20260926 23:21:29.684856 4056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:29.685877 4056 log.cc:824] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:29.687229 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:29.675877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.16.62.126:44789)
0926 23:21:29.676822 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:29.676832 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:29.676871 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:21:29.678401 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:29.678421 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:29.678875 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:29.679260 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:29.679839 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:29.681118 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:29.682456 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:29.683122 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:29.683332 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:29.684523 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:29.684530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:29.684543 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:29.685099 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:29.685190 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:29.685213 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:29.685356 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:29.685537 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 23:21:29.685943 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":358,"threads_started":1}
I20260926 23:21:29.689186 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1: No bootstrap required, opened a new log
I20260926 23:21:29.696605 4056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e20a28b4d24105b63ccef77dff84c1" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44789 } }
I20260926 23:21:29.696920 4056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:29.696988 4056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e20a28b4d24105b63ccef77dff84c1, State: Initialized, Role: FOLLOWER
I20260926 23:21:29.697506 4056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e20a28b4d24105b63ccef77dff84c1" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44789 } }
I20260926 23:21:29.697703 4056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:29.697793 4056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:29.697904 4056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:29.699899 4056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e20a28b4d24105b63ccef77dff84c1" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44789 } }
I20260926 23:21:29.700379 4056 leader_election.cc:302] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44e20a28b4d24105b63ccef77dff84c1; no voters:
I20260926 23:21:29.700852 4056 leader_election.cc:288] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:29.701059 4061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:29.701913 4061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [term 1 LEADER]: Becoming Leader. State: Replica: 44e20a28b4d24105b63ccef77dff84c1, State: Running, Role: LEADER
I20260926 23:21:29.702440 4061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e20a28b4d24105b63ccef77dff84c1" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44789 } }
I20260926 23:21:29.702950 4056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:29.705258 4063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44e20a28b4d24105b63ccef77dff84c1. Latest consensus state: current_term: 1 leader_uuid: "44e20a28b4d24105b63ccef77dff84c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e20a28b4d24105b63ccef77dff84c1" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44789 } } }
I20260926 23:21:29.705741 4063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:29.706542 4062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44e20a28b4d24105b63ccef77dff84c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e20a28b4d24105b63ccef77dff84c1" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 44789 } } }
I20260926 23:21:29.706902 4062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:29.708304 4070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:29.713176 4070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:29.722210 4070 catalog_manager.cc:1347] Generated new cluster ID: 7f74e034c46645429222c9a04462cf75
I20260926 23:21:29.722329 4070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:29.730376 4070 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:29.731503 4070 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:29.737466 4070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1: Generated new TSK 0
I20260926 23:21:29.738205 4070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:29.751168 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.65:0
--local_ip_for_outbound_sockets=127.16.62.65
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44789
--builtin_ntp_servers=127.16.62.84:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:21:30.094573 4080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:30.094872 4080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:30.095028 4080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:30.102771 4080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:30.102895 4080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:30.103107 4080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.65
I20260926 23:21:30.112345 4080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.62.65
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.65
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:30.113543 4080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:30.115247 4080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:30.128013 4086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:30.128099 4088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:30.128170 4085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:30.128913 4080 server_base.cc:1034] running on GCE node
I20260926 23:21:30.129456 4080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:30.130348 4080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:30.131574 4080 hybrid_clock.cc:648] HybridClock initialized: now 1790464890131544 us; error 49 us; skew 500 ppm
I20260926 23:21:30.131870 4080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:30.133961 4080 webserver.cc:460] Webserver started at http://127.16.62.65:43613/ using document root <none> and password file <none>
I20260926 23:21:30.134621 4080 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:30.134770 4080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:30.135092 4080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:30.138000 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/data/instance:
uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8"
format_stamp: "Formatted at 2026-09-26 23:21:30 on dist-test-slave-vcv1"
server_key: "8ef4fcfdd88a559aaefa5d0d2d8fbbff"
server_key_iv: "9887b31244e41488c2e341e82d7d50f1"
server_key_version: "encryptionkey@0"
I20260926 23:21:30.138830 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance:
uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8"
format_stamp: "Formatted at 2026-09-26 23:21:30 on dist-test-slave-vcv1"
server_key: "8ef4fcfdd88a559aaefa5d0d2d8fbbff"
server_key_iv: "9887b31244e41488c2e341e82d7d50f1"
server_key_version: "encryptionkey@0"
I20260926 23:21:30.143636 4080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 23:21:30.147037 4094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:30.148314 4080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 23:21:30.148485 4080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/wal
uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8"
format_stamp: "Formatted at 2026-09-26 23:21:30 on dist-test-slave-vcv1"
server_key: "8ef4fcfdd88a559aaefa5d0d2d8fbbff"
server_key_iv: "9887b31244e41488c2e341e82d7d50f1"
server_key_version: "encryptionkey@0"
I20260926 23:21:30.148694 4080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:30.174671 4080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:30.175472 4080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:30.175791 4080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:30.176789 4080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:30.178184 4080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:30.178303 4080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:30.178443 4080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:30.178530 4080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:30.211450 4080 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.65:39611
I20260926 23:21:30.211536 4206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.65:39611 every 8 connection(s)
I20260926 23:21:30.212805 4080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/data/info.pb
I20260926 23:21:30.219842 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 4080
I20260926 23:21:30.220027 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/wal/instance
I20260926 23:21:30.220489 16633 external_mini_cluster.cc:1428] Setting key a4ded6d7f2a07fb084d0772707a591d5
I20260926 23:21:30.223212 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.66:0
--local_ip_for_outbound_sockets=127.16.62.66
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44789
--builtin_ntp_servers=127.16.62.84:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:21:30.225397 4209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:30.215166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:46345 (local address 127.16.62.126:44789)
0926 23:21:30.215595 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:30.215599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:30.216684 (+ 1085us) server_negotiation.cc:407] Connection header received
0926 23:21:30.218596 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:30.218602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:30.218689 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:30.218784 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:30.219781 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:30.220659 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:30.222473 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:30.223003 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:30.223166 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:30.224560 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:30.224566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:30.224571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:30.224598 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:30.224639 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:30.224651 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:30.224743 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:30.224959 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 23:21:30.225051 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":237,"threads_started":1}
I20260926 23:21:30.226337 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:30.215606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44789 (local address 127.16.62.65:46345)
0926 23:21:30.216524 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:30.216559 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:30.218263 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:30.218982 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:30.218989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:30.219070 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:30.219608 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:30.219627 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:30.220814 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:30.220819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:30.222328 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:30.222339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:30.223236 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:30.223245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:30.223547 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:30.224261 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:21:30.224264 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:30.224305 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:21:30.224380 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:30.224743 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:30.224756 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:30.224766 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:21:30.224950 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 23:21:30.225314 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":245,"threads_started":1}
I20260926 23:21:30.227893 4207 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44789
I20260926 23:21:30.228242 4207 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:30.228915 4207 heartbeater.cc:511] Master 127.16.62.126:44789 requested a full tablet report, sending...
I20260926 23:21:30.230909 4021 ts_manager.cc:194] Registered new tserver with Master: e9a3a0111c9b4e0b8930204cf47cf6c8 (127.16.62.65:39611)
I20260926 23:21:30.232741 4021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.65:46345
W20260926 23:21:30.543175 4211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:30.543455 4211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:30.543604 4211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:30.550813 4211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:30.550902 4211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:30.551047 4211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.66
I20260926 23:21:30.559873 4211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.62.66
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.66
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:30.561144 4211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:30.562848 4211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:30.576083 4216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:30.576188 4219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:30.576115 4217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:30.576994 4211 server_base.cc:1034] running on GCE node
I20260926 23:21:30.577608 4211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:30.578496 4211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:30.579725 4211 hybrid_clock.cc:648] HybridClock initialized: now 1790464890579690 us; error 48 us; skew 500 ppm
I20260926 23:21:30.580037 4211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:30.582139 4211 webserver.cc:460] Webserver started at http://127.16.62.66:35431/ using document root <none> and password file <none>
I20260926 23:21:30.582820 4211 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:30.582950 4211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:30.583284 4211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:30.586057 4211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/data/instance:
uuid: "2b5e04f59b1f4f729685906f1fe89608"
format_stamp: "Formatted at 2026-09-26 23:21:30 on dist-test-slave-vcv1"
server_key: "7fd062bfcb4e3279501f88853d626165"
server_key_iv: "cfdab12d79163e4fe761fb4f2a6ed735"
server_key_version: "encryptionkey@0"
I20260926 23:21:30.586851 4211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance:
uuid: "2b5e04f59b1f4f729685906f1fe89608"
format_stamp: "Formatted at 2026-09-26 23:21:30 on dist-test-slave-vcv1"
server_key: "7fd062bfcb4e3279501f88853d626165"
server_key_iv: "cfdab12d79163e4fe761fb4f2a6ed735"
server_key_version: "encryptionkey@0"
I20260926 23:21:30.591701 4211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 23:21:30.595074 4225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:30.596397 4211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:30.596575 4211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/wal
uuid: "2b5e04f59b1f4f729685906f1fe89608"
format_stamp: "Formatted at 2026-09-26 23:21:30 on dist-test-slave-vcv1"
server_key: "7fd062bfcb4e3279501f88853d626165"
server_key_iv: "cfdab12d79163e4fe761fb4f2a6ed735"
server_key_version: "encryptionkey@0"
I20260926 23:21:30.596786 4211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:30.614884 4211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:30.615635 4211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:30.615955 4211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:30.616931 4211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:30.618412 4211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:30.618546 4211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:30.618678 4211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:30.618777 4211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:30.651693 4211 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.66:34973
I20260926 23:21:30.651710 4337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.66:34973 every 8 connection(s)
I20260926 23:21:30.653205 4211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/data/info.pb
I20260926 23:21:30.661607 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 4211
I20260926 23:21:30.661886 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/wal/instance
I20260926 23:21:30.662520 16633 external_mini_cluster.cc:1428] Setting key 55fa4895e16418537a35a2af17484b4f
I20260926 23:21:30.665913 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.62.67:0
--local_ip_for_outbound_sockets=127.16.62.67
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44789
--builtin_ntp_servers=127.16.62.84:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:21:30.672610 4209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:30.655595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.66:56545 (local address 127.16.62.126:44789)
0926 23:21:30.655775 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:30.655780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:30.663034 (+ 7254us) server_negotiation.cc:407] Connection header received
0926 23:21:30.665577 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:30.665583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:30.665692 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:30.665816 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:30.667057 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:30.667886 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:30.669639 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:30.670085 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:30.670233 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:30.671692 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:30.671700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:30.671711 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:30.671731 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:30.671763 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:30.671775 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:30.671877 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:30.672122 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 23:21:30.672263 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:21:30.673779 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:30.656056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44789 (local address 127.16.62.66:56545)
0926 23:21:30.662827 (+ 6771us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:30.662873 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:30.665275 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:30.666118 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:30.666133 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:30.666273 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:30.666880 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:30.666899 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:30.668064 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:30.668069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:30.669479 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:30.669493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:30.670332 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:30.670340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:30.670579 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:30.671364 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:21:30.671367 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:30.671408 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:21:30.671492 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:30.671877 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:30.671891 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:30.671902 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:21:30.672095 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 23:21:30.672552 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6155,"thread_start_us":5894,"threads_started":1}
I20260926 23:21:30.675297 4338 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44789
I20260926 23:21:30.675635 4338 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:30.676491 4338 heartbeater.cc:511] Master 127.16.62.126:44789 requested a full tablet report, sending...
I20260926 23:21:30.678045 4021 ts_manager.cc:194] Registered new tserver with Master: 2b5e04f59b1f4f729685906f1fe89608 (127.16.62.66:34973)
I20260926 23:21:30.679448 4021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.66:56545
W20260926 23:21:31.005710 4341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:31.006008 4341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:31.006181 4341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:31.013885 4341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:31.014021 4341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:31.014226 4341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.62.67
I20260926 23:21:31.023584 4341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.62.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.62.67
--webserver_port=0
--tserver_master_addrs=127.16.62.126:44789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.62.67
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:31.025104 4341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:31.026989 4341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:31.040620 4346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:31.040634 4349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:31.040647 4347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:31.041399 4341 server_base.cc:1034] running on GCE node
I20260926 23:21:31.042007 4341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:31.042865 4341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:31.044102 4341 hybrid_clock.cc:648] HybridClock initialized: now 1790464891044033 us; error 53 us; skew 500 ppm
I20260926 23:21:31.044478 4341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:21:31.046864 4341 webserver.cc:460] Webserver started at http://127.16.62.67:38169/ using document root <none> and password file <none>
I20260926 23:21:31.047559 4341 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:31.047693 4341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:31.048027 4341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:31.050851 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/data/instance:
uuid: "00a388f696ac4466a5327e09d558df5b"
format_stamp: "Formatted at 2026-09-26 23:21:31 on dist-test-slave-vcv1"
server_key: "124fb946f7d1259c055135fe102abc5f"
server_key_iv: "4d0756274e69cbfb90338fc7cf3d9528"
server_key_version: "encryptionkey@0"
I20260926 23:21:31.051702 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance:
uuid: "00a388f696ac4466a5327e09d558df5b"
format_stamp: "Formatted at 2026-09-26 23:21:31 on dist-test-slave-vcv1"
server_key: "124fb946f7d1259c055135fe102abc5f"
server_key_iv: "4d0756274e69cbfb90338fc7cf3d9528"
server_key_version: "encryptionkey@0"
I20260926 23:21:31.056898 4341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:21:31.060360 4355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:31.061597 4341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:21:31.061791 4341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/wal
uuid: "00a388f696ac4466a5327e09d558df5b"
format_stamp: "Formatted at 2026-09-26 23:21:31 on dist-test-slave-vcv1"
server_key: "124fb946f7d1259c055135fe102abc5f"
server_key_iv: "4d0756274e69cbfb90338fc7cf3d9528"
server_key_version: "encryptionkey@0"
I20260926 23:21:31.062023 4341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:31.103322 4341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:31.104278 4341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:31.104599 4341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:31.105645 4341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:21:31.107177 4341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:21:31.107298 4341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:31.107416 4341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:21:31.107507 4341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:31.147830 4341 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.67:33125
I20260926 23:21:31.147872 4467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.67:33125 every 8 connection(s)
I20260926 23:21:31.149201 4341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/data/info.pb
I20260926 23:21:31.156499 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 4341
I20260926 23:21:31.156687 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/wal/instance
I20260926 23:21:31.157114 16633 external_mini_cluster.cc:1428] Setting key 3865936cddfb0fb62f7b1fd43a009675
I20260926 23:21:31.165657 4209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.152072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.67:34455 (local address 127.16.62.126:44789)
0926 23:21:31.152307 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.152314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.154044 (+ 1730us) server_negotiation.cc:407] Connection header received
0926 23:21:31.157312 (+ 3268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.157318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.157406 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.157525 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:31.159140 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.160551 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.162117 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.163024 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.163223 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.164584 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:31.164594 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:31.164602 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:31.164635 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:31.164686 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:31.164710 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:31.164835 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.165034 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.165172 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 23:21:31.166311 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.152675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:44789 (local address 127.16.62.67:34455)
0926 23:21:31.153830 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.153880 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:31.156902 (+ 3022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:31.157834 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:31.157849 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:31.157998 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:21:31.158891 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.158931 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.160705 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.160709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.161958 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.161970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.163260 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.163265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.163456 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.164234 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:21:31.164237 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:21:31.164274 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:21:31.164354 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:21:31.164823 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:21:31.164831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:21:31.164836 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:21:31.164974 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 23:21:31.165276 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":315,"threads_started":1}
I20260926 23:21:31.167694 4468 heartbeater.cc:347] Connected to a master server at 127.16.62.126:44789
I20260926 23:21:31.168005 4468 heartbeater.cc:464] Registering TS with master...
I20260926 23:21:31.168969 4468 heartbeater.cc:511] Master 127.16.62.126:44789 requested a full tablet report, sending...
I20260926 23:21:31.170526 4021 ts_manager.cc:194] Registered new tserver with Master: 00a388f696ac4466a5327e09d558df5b (127.16.62.67:33125)
I20260926 23:21:31.171736 4021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.62.67:34455
I20260926 23:21:31.173033 16633 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:21:31.185261 4209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.177933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.16.62.126:44789)
0926 23:21:31.178202 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.178212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.178722 (+ 510us) server_negotiation.cc:407] Connection header received
0926 23:21:31.178855 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.178863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.178969 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.179090 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:31.179562 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.180893 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.182284 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.183168 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.183326 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.184328 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:31.184338 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:31.184346 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:31.184372 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:31.184414 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:31.184434 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:31.184556 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.184646 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.184777 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:21:31.189783 4021 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:21:31.204000 4021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:21:31.235402 4207 heartbeater.cc:503] Master 127.16.62.126:44789 was elected leader, sending a full tablet report...
I20260926 23:21:31.239683 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.218508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:34973 (local address 127.0.0.1:57930)
0926 23:21:31.225681 (+ 7173us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.225727 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:31.225936 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:31.229755 (+ 3819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:31.229763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:31.229794 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:31.230240 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.230253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.232669 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.232677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.235360 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.235377 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.238584 (+ 3207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.238590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.238835 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.238908 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 23:21:31.239025 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 23:21:31.239234 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6858,"mutex_wait_us":112,"thread_start_us":975,"threads_started":1}
I20260926 23:21:31.241394 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.218644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.16.62.66:34973)
0926 23:21:31.229065 (+ 10421us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.229074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.229108 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:21:31.229212 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.229219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.229473 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.230439 (+ 966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:31.230686 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.232490 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.236878 (+ 4388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.238326 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.240461 (+ 2135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.240733 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.240813 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.241002 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10253,"thread_start_us":10155,"threads_started":1}
I20260926 23:21:31.241875 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.220344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:33125 (local address 127.0.0.1:41046)
0926 23:21:31.229866 (+ 9522us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.229883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:31.229989 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:31.230661 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:31.230665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:31.230691 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:31.231149 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.231163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.234239 (+ 3076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.234247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.237479 (+ 3232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.237497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.240975 (+ 3478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.240982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.241177 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.241214 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:21:31.241314 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 23:21:31.241464 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9393,"mutex_wait_us":3,"thread_start_us":9816,"threads_started":1}
I20260926 23:21:31.242646 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.219078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:39611 (local address 127.0.0.1:40370)
0926 23:21:31.229149 (+ 10071us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.229168 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:31.229299 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:31.230222 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:31.230226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:31.230252 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:31.230661 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.230673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.232779 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.232789 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.238351 (+ 5562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.238366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.241594 (+ 3228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.241603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.241867 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.241917 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 23:21:31.242018 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 23:21:31.242180 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9924,"mutex_wait_us":1095,"thread_start_us":9763,"threads_started":1}
I20260926 23:21:31.242746 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.220906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.16.62.67:33125)
0926 23:21:31.229440 (+ 8534us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.229449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.230118 (+ 669us) server_negotiation.cc:407] Connection header received
0926 23:21:31.230241 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.230248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.230448 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.230627 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:31.231390 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.234042 (+ 2652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.237736 (+ 3694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.240439 (+ 2703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.240573 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.240818 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.242186 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.242349 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8356,"thread_start_us":8300,"threads_started":1}
I20260926 23:21:31.244093 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.219358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.16.62.65:39611)
0926 23:21:31.228777 (+ 9419us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.228799 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.229439 (+ 640us) server_negotiation.cc:407] Connection header received
0926 23:21:31.229557 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.229565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.229765 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.229955 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:31.230909 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.232584 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.238601 (+ 6017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.240967 (+ 2366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.241125 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.241444 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.242924 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.243652 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9196,"thread_start_us":9050,"threads_started":1}
I20260926 23:21:31.245766 4273 tablet_service.cc:1467] Processing CreateTablet for tablet 895c4a77516f4faab02347a168c302e2 (DEFAULT_TABLE table=test-table [id=64544c07a1d34e998068d1581c65250f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:31.247615 4273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895c4a77516f4faab02347a168c302e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:31.247705 4142 tablet_service.cc:1467] Processing CreateTablet for tablet 895c4a77516f4faab02347a168c302e2 (DEFAULT_TABLE table=test-table [id=64544c07a1d34e998068d1581c65250f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:31.248337 4403 tablet_service.cc:1467] Processing CreateTablet for tablet 895c4a77516f4faab02347a168c302e2 (DEFAULT_TABLE table=test-table [id=64544c07a1d34e998068d1581c65250f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:21:31.249795 4403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895c4a77516f4faab02347a168c302e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:31.250380 4142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895c4a77516f4faab02347a168c302e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:31.269546 4486 tablet_bootstrap.cc:492] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: Bootstrap starting.
I20260926 23:21:31.276902 4486 tablet_bootstrap.cc:654] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:31.278044 4487 tablet_bootstrap.cc:492] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: Bootstrap starting.
I20260926 23:21:31.280506 4488 tablet_bootstrap.cc:492] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: Bootstrap starting.
I20260926 23:21:31.282485 4487 tablet_bootstrap.cc:654] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:31.283558 4487 log.cc:824] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:31.284577 4488 tablet_bootstrap.cc:654] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:31.285889 4488 log.cc:824] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:31.286686 4487 tablet_bootstrap.cc:492] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: No bootstrap required, opened a new log
I20260926 23:21:31.287016 4487 ts_tablet_manager.cc:1397] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260926 23:21:31.287345 4486 log.cc:824] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:31.295281 4488 tablet_bootstrap.cc:492] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: No bootstrap required, opened a new log
I20260926 23:21:31.295720 4488 ts_tablet_manager.cc:1397] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.004s
I20260926 23:21:31.302282 4487 raft_consensus.cc:348] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } }
I20260926 23:21:31.302861 4486 tablet_bootstrap.cc:492] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: No bootstrap required, opened a new log
I20260926 23:21:31.302911 4487 raft_consensus.cc:374] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:31.303085 4487 raft_consensus.cc:729] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a388f696ac4466a5327e09d558df5b, State: Initialized, Role: FOLLOWER
I20260926 23:21:31.303208 4486 ts_tablet_manager.cc:1397] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.007s
I20260926 23:21:31.304046 4487 consensus_queue.cc:260] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } }
I20260926 23:21:31.309135 4488 raft_consensus.cc:348] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } }
I20260926 23:21:31.309676 4488 raft_consensus.cc:374] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:31.309819 4488 raft_consensus.cc:729] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a3a0111c9b4e0b8930204cf47cf6c8, State: Initialized, Role: FOLLOWER
I20260926 23:21:31.310577 4488 consensus_queue.cc:260] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } }
I20260926 23:21:31.311331 4468 heartbeater.cc:503] Master 127.16.62.126:44789 was elected leader, sending a full tablet report...
I20260926 23:21:31.312348 4487 ts_tablet_manager.cc:1428] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: Time spent starting tablet: real 0.025s user 0.019s sys 0.004s
I20260926 23:21:31.316483 4486 raft_consensus.cc:348] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } }
I20260926 23:21:31.316999 4486 raft_consensus.cc:374] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:31.317149 4486 raft_consensus.cc:729] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5e04f59b1f4f729685906f1fe89608, State: Initialized, Role: FOLLOWER
I20260926 23:21:31.317854 4486 consensus_queue.cc:260] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } }
I20260926 23:21:31.318760 4488 ts_tablet_manager.cc:1428] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: Time spent starting tablet: real 0.023s user 0.017s sys 0.003s
I20260926 23:21:31.318929 4338 heartbeater.cc:503] Master 127.16.62.126:44789 was elected leader, sending a full tablet report...
I20260926 23:21:31.319528 4486 ts_tablet_manager.cc:1428] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
W20260926 23:21:31.401224 4469 tablet.cc:2369] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:21:31.405093 4339 tablet.cc:2369] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:21:31.436506 4493 raft_consensus.cc:482] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:21:31.436811 4493 raft_consensus.cc:504] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } }
I20260926 23:21:31.438469 4493 leader_election.cc:288] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b5e04f59b1f4f729685906f1fe89608 (127.16.62.66:34973), 00a388f696ac4466a5327e09d558df5b (127.16.62.67:33125)
I20260926 23:21:31.444851 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.438914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:39459 (local address 127.16.62.66:34973)
0926 23:21:31.439098 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.439102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.439673 (+ 571us) server_negotiation.cc:407] Connection header received
0926 23:21:31.439882 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.439890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.439985 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.440095 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:31.440781 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.441470 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.443207 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.443960 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.444165 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.444238 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.444472 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.444594 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:21:31.444898 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.438866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.66:34973 (local address 127.16.62.65:39459)
0926 23:21:31.439566 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.439587 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:31.439733 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:31.440235 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:31.440241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:31.440267 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:31.440649 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.440659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.441605 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.441611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.443029 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.443040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.444160 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.444167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.444323 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.444345 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:21:31.444449 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 23:21:31.444570 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":388,"threads_started":1}
I20260926 23:21:31.445333 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.439857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.62.65:49415 (local address 127.16.62.67:33125)
0926 23:21:31.440045 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.440071 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.440469 (+ 398us) server_negotiation.cc:407] Connection header received
0926 23:21:31.440605 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.440609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.440666 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.440764 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:21:31.441311 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.442005 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.443682 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.444490 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.444648 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.444717 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.444987 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.445106 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:21:31.445399 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.439719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.67:33125 (local address 127.16.62.65:49415)
0926 23:21:31.440332 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.440355 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:21:31.440487 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:21:31.440797 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:21:31.440801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:21:31.440825 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:21:31.441137 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.441146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.442139 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.442150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.443504 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:21:31.443516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.444684 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.444691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:21:31.444836 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.444857 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 23:21:31.444962 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 23:21:31.445079 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":1,"thread_start_us":283,"threads_started":1}
I20260926 23:21:31.445955 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895c4a77516f4faab02347a168c302e2" candidate_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b5e04f59b1f4f729685906f1fe89608" is_pre_election: true
I20260926 23:21:31.446138 4423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895c4a77516f4faab02347a168c302e2" candidate_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a388f696ac4466a5327e09d558df5b" is_pre_election: true
I20260926 23:21:31.446491 4293 raft_consensus.cc:2454] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a3a0111c9b4e0b8930204cf47cf6c8 in term 0.
I20260926 23:21:31.446660 4423 raft_consensus.cc:2454] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a3a0111c9b4e0b8930204cf47cf6c8 in term 0.
I20260926 23:21:31.447145 4097 leader_election.cc:302] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b5e04f59b1f4f729685906f1fe89608, e9a3a0111c9b4e0b8930204cf47cf6c8; no voters:
I20260926 23:21:31.447508 4493 raft_consensus.cc:2781] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:21:31.447649 4493 raft_consensus.cc:482] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:21:31.447789 4493 raft_consensus.cc:3037] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:31.449949 4493 raft_consensus.cc:504] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } }
I20260926 23:21:31.450510 4493 leader_election.cc:288] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [CANDIDATE]: Term 1 election: Requested vote from peers 2b5e04f59b1f4f729685906f1fe89608 (127.16.62.66:34973), 00a388f696ac4466a5327e09d558df5b (127.16.62.67:33125)
I20260926 23:21:31.450904 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895c4a77516f4faab02347a168c302e2" candidate_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b5e04f59b1f4f729685906f1fe89608"
I20260926 23:21:31.451051 4423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895c4a77516f4faab02347a168c302e2" candidate_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a388f696ac4466a5327e09d558df5b"
I20260926 23:21:31.451097 4293 raft_consensus.cc:3037] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:31.451249 4423 raft_consensus.cc:3037] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:31.453501 4293 raft_consensus.cc:2454] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a3a0111c9b4e0b8930204cf47cf6c8 in term 1.
I20260926 23:21:31.453545 4423 raft_consensus.cc:2454] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a3a0111c9b4e0b8930204cf47cf6c8 in term 1.
I20260926 23:21:31.453876 4097 leader_election.cc:302] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b5e04f59b1f4f729685906f1fe89608, e9a3a0111c9b4e0b8930204cf47cf6c8; no voters:
I20260926 23:21:31.454176 4493 raft_consensus.cc:2781] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:31.454592 4493 raft_consensus.cc:686] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 1 LEADER]: Becoming Leader. State: Replica: e9a3a0111c9b4e0b8930204cf47cf6c8, State: Running, Role: LEADER
I20260926 23:21:31.455036 4493 consensus_queue.cc:237] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } }
I20260926 23:21:31.459254 4020 catalog_manager.cc:5495] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a3a0111c9b4e0b8930204cf47cf6c8 (127.16.62.65). New cstate: current_term: 1 leader_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a3a0111c9b4e0b8930204cf47cf6c8" member_type: VOTER last_known_addr { host: "127.16.62.65" port: 39611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 } health_report { overall_health: UNKNOWN } } }
W20260926 23:21:31.465179 4208 tablet.cc:2369] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:21:31.531026 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.526166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48418 (local address 127.16.62.65:39611)
0926 23:21:31.526336 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.526341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.526356 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:21:31.526456 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.526460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.526539 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.526711 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:21:31.527077 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.527810 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.529200 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.529587 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.529675 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.530091 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:21:31.530592 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:21:31.530698 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.530775 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.530836 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:21:31.538853 4423 raft_consensus.cc:1261] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [term 1 FOLLOWER]: Refusing update from remote peer e9a3a0111c9b4e0b8930204cf47cf6c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:31.538856 4293 raft_consensus.cc:1261] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [term 1 FOLLOWER]: Refusing update from remote peer e9a3a0111c9b4e0b8930204cf47cf6c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:21:31.539794 4497 consensus_queue.cc:1035] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b5e04f59b1f4f729685906f1fe89608" member_type: VOTER last_known_addr { host: "127.16.62.66" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:31.540206 4493 consensus_queue.cc:1035] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a388f696ac4466a5327e09d558df5b" member_type: VOTER last_known_addr { host: "127.16.62.67" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:21:31.562636 4021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:31.562932 4021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:21:31.566542 4021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44e20a28b4d24105b63ccef77dff84c1: Sending DeleteTablet for 3 replicas of tablet 895c4a77516f4faab02347a168c302e2
I20260926 23:21:31.567620 4273 tablet_service.cc:1514] Processing DeleteTablet for tablet 895c4a77516f4faab02347a168c302e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:31 UTC) from {username='slave'} at 127.0.0.1:57930
I20260926 23:21:31.567668 4403 tablet_service.cc:1514] Processing DeleteTablet for tablet 895c4a77516f4faab02347a168c302e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:31 UTC) from {username='slave'} at 127.0.0.1:41046
I20260926 23:21:31.567698 4142 tablet_service.cc:1514] Processing DeleteTablet for tablet 895c4a77516f4faab02347a168c302e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:21:31 UTC) from {username='slave'} at 127.0.0.1:40370
I20260926 23:21:31.574311 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4080
I20260926 23:21:31.583565 4510 tablet_replica.cc:330] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: stopping tablet replica
I20260926 23:21:31.583982 4511 tablet_replica.cc:330] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: stopping tablet replica
I20260926 23:21:31.584688 4511 raft_consensus.cc:2229] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:21:31.585508 4511 raft_consensus.cc:2258] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:31.586313 4509 tablet_replica.cc:330] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: stopping tablet replica
I20260926 23:21:31.586356 4510 raft_consensus.cc:2229] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:31.586889 4510 raft_consensus.cc:2258] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:31.587154 4509 raft_consensus.cc:2229] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:21:31.587667 4509 raft_consensus.cc:2258] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:21:31.588698 4511 ts_tablet_manager.cc:1905] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:31.589299 4510 ts_tablet_manager.cc:1905] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:31.589570 4509 ts_tablet_manager.cc:1905] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:21:31.593359 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.576011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.16.62.65:39611)
0926 23:21:31.576212 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.576219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.576643 (+ 424us) server_negotiation.cc:407] Connection header received
0926 23:21:31.577433 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.577444 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.577576 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.577654 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:31.578807 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.580047 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.584271 (+ 4224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.584974 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.585104 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.591652 (+ 6548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:31.591664 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:31.591680 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:31.592429 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:31.592490 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:31.592514 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:31.592598 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.592855 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.592960 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:21:31.594761 4202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:31.596549 4509 ts_tablet_manager.cc:1918] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:31.596733 4509 log.cc:1196] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-1/wal/wals/895c4a77516f4faab02347a168c302e2
I20260926 23:21:31.597242 4509 ts_tablet_manager.cc:1939] T 895c4a77516f4faab02347a168c302e2 P 2b5e04f59b1f4f729685906f1fe89608: Deleting consensus metadata
I20260926 23:21:31.598773 4007 catalog_manager.cc:4841] TS 2b5e04f59b1f4f729685906f1fe89608 (127.16.62.66:34973): tablet 895c4a77516f4faab02347a168c302e2 (table test-table [id=64544c07a1d34e998068d1581c65250f]) successfully deleted
I20260926 23:21:31.601217 4510 ts_tablet_manager.cc:1918] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:31.601385 4510 log.cc:1196] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-2/wal/wals/895c4a77516f4faab02347a168c302e2
I20260926 23:21:31.601938 4510 ts_tablet_manager.cc:1939] T 895c4a77516f4faab02347a168c302e2 P 00a388f696ac4466a5327e09d558df5b: Deleting consensus metadata
I20260926 23:21:31.603819 4007 catalog_manager.cc:4841] TS 00a388f696ac4466a5327e09d558df5b (127.16.62.67:33125): tablet 895c4a77516f4faab02347a168c302e2 (table test-table [id=64544c07a1d34e998068d1581c65250f]) successfully deleted
I20260926 23:21:31.699589 4511 ts_tablet_manager.cc:1918] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:21:31.699874 4511 log.cc:1196] T 895c4a77516f4faab02347a168c302e2 P e9a3a0111c9b4e0b8930204cf47cf6c8: Deleting WAL directory at /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464575313384-16633-0/minicluster-data/ts-0/wal/wals/895c4a77516f4faab02347a168c302e2
I20260926 23:21:31.700372 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 4080
I20260926 23:21:31.718550 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4211
W20260926 23:21:31.718559 4008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.62.65:39611 (error 108)
I20260926 23:21:31.720306 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.719820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:39611 (local address 127.0.0.1:48434)
0926 23:21:31.720012 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.720104 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 23:21:31.720582 4008 catalog_manager.cc:4584] TS e9a3a0111c9b4e0b8930204cf47cf6c8 (127.16.62.65:39611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 895c4a77516f4faab02347a168c302e2: Network error: Client connection negotiation failed: client connection to 127.16.62.65:39611: connect: Connection refused (error 111)
I20260926 23:21:31.726035 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.719807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.16.62.66:34973)
0926 23:21:31.720021 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.720030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.720049 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:21:31.720116 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.720119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.720287 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.720402 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:31.720812 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.721649 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.723109 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.723517 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.723647 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.724724 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:31.724732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:31.724744 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:31.725314 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:31.725359 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:31.725378 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:31.725470 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.725554 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.725636 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:21:31.726648 4333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:31.774240 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.773687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:39611 (local address 127.0.0.1:48444)
0926 23:21:31.773919 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.774047 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 23:21:31.775354 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.774970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:39611 (local address 127.0.0.1:48454)
0926 23:21:31.775123 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.775205 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 23:21:31.823467 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 4211
I20260926 23:21:31.835855 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.835226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:39611 (local address 127.0.0.1:48470)
0926 23:21:31.835482 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.835601 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 23:21:31.837028 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.836586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:39611 (local address 127.0.0.1:48484)
0926 23:21:31.836773 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.836870 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 23:21:31.840978 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4341
I20260926 23:21:31.848351 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.841717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.16.62.67:33125)
0926 23:21:31.841971 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:31.841977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:31.842000 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:21:31.842144 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:31.842148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:31.842242 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:31.842373 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:31.842821 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.843574 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.845016 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:31.845440 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:31.845598 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:31.846739 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:31.846749 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:31.846763 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:31.847532 (+ 769us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:31.847578 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:31.847599 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:31.847744 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:31.847877 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 23:21:31.848004 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:21:31.848944 4463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:31.917613 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.917058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:39611 (local address 127.0.0.1:48500)
0926 23:21:31.917353 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.917446 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260926 23:21:31.918771 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:31.918327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.65:39611 (local address 127.0.0.1:48514)
0926 23:21:31.918569 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 23:21:31.918643 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.65:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260926 23:21:31.937556 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 4341
I20260926 23:21:31.953969 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 3990
I20260926 23:21:31.955122 4051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:32.064051 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 3990
2026-09-26T23:21:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2877 ms)
[----------] 6 tests from EncryptionPolicyTest (21293 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:32 dist-test-slave-vcv1 krb5kdc[4520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:32 dist-test-slave-vcv1 krb5kdc[4520](info): set up 2 sockets
Sep 26 23:21:32 dist-test-slave-vcv1 krb5kdc[4520](info): commencing operation
krb5kdc: starting...
W20260926 23:21:33.185568 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:33 dist-test-slave-vcv1 krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:33.427786 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:33.756862 4533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:33.757164 4533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:33.757320 4533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:33.764662 4533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:33.764808 4533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:33.764954 4533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:33.765054 4533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:33.765120 4533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:33.775786 4533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:33.777078 4533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:33.778647 4533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:33.790624 4538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:33.790685 4541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:33.790657 4539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:33.791311 4533 server_base.cc:1034] running on GCE node
I20260926 23:21:33.792116 4533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:33.793515 4533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:33.794752 4533 hybrid_clock.cc:648] HybridClock initialized: now 1790464893794685 us; error 48 us; skew 500 ppm
I20260926 23:21:33.795158 4533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:33 dist-test-slave-vcv1 krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:33.798498 4533 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:33.799995 4533 webserver.cc:460] Webserver started at http://127.0.0.1:41413/ using document root <none> and password file <none>
I20260926 23:21:33.800740 4533 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:33.800887 4533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:33.801234 4533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:33.804369 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "cb778469275943bc8da301a820439caf"
format_stamp: "Formatted at 2026-09-26 23:21:33 on dist-test-slave-vcv1"
server_key: "2dc985a35b56be4c9851574b9e4cb530"
server_key_iv: "4e73b8cf11067d7099d4e49dabff043b"
server_key_version: "encryptionkey@0"
I20260926 23:21:33.805178 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "cb778469275943bc8da301a820439caf"
format_stamp: "Formatted at 2026-09-26 23:21:33 on dist-test-slave-vcv1"
server_key: "2dc985a35b56be4c9851574b9e4cb530"
server_key_iv: "4e73b8cf11067d7099d4e49dabff043b"
server_key_version: "encryptionkey@0"
I20260926 23:21:33.810434 4533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:21:33.813793 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:33.815112 4533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:21:33.815306 4533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "cb778469275943bc8da301a820439caf"
format_stamp: "Formatted at 2026-09-26 23:21:33 on dist-test-slave-vcv1"
server_key: "2dc985a35b56be4c9851574b9e4cb530"
server_key_iv: "4e73b8cf11067d7099d4e49dabff043b"
server_key_version: "encryptionkey@0"
I20260926 23:21:33.815519 4533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:33.844719 4533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:33.849205 4533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:33.849527 4533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:33.869244 4599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38017 every 8 connection(s)
I20260926 23:21:33.869241 4533 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38017
I20260926 23:21:33.870775 4533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:33.874384 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:33.881064 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 4533
I20260926 23:21:33.881273 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:33.881934 16633 external_mini_cluster.cc:1428] Setting key 07e3af89717c9466b27b7d61b4669f1a
I20260926 23:21:33.884629 4600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf: Bootstrap starting.
I20260926 23:21:33.887611 4600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:33.888690 4600 log.cc:824] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:33.891368 4600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf: No bootstrap required, opened a new log
Sep 26 23:21:33 dist-test-slave-vcv1 krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:21:33.899706 4600 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb778469275943bc8da301a820439caf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } }
I20260926 23:21:33.900096 4600 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:33.900245 4600 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb778469275943bc8da301a820439caf, State: Initialized, Role: FOLLOWER
I20260926 23:21:33.900903 4600 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb778469275943bc8da301a820439caf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } }
I20260926 23:21:33.901158 4600 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:33.901306 4600 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:33.901487 4600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:33.903956 4600 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb778469275943bc8da301a820439caf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } }
I20260926 23:21:33.904501 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:33.883798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46921 (local address 127.0.0.1:38017)
0926 23:21:33.884591 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:33.884602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:33.884655 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:21:33.886375 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:33.886398 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:33.886765 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:33.887124 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:33.888687 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:33.890384 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:33.891670 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:33.892390 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:33.892602 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:33.897058 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:33.897066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:33.897081 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:33.897124 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:33.900174 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:33.900961 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:33.900974 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:33.900981 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:33.901181 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:33.901744 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:33.901749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:33.901752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:33.902288 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:33.902623 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:33.903112 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 23:21:33.903430 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":389,"threads_started":1}
I20260926 23:21:33.904506 4600 leader_election.cc:302] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb778469275943bc8da301a820439caf; no voters:
I20260926 23:21:33.905282 4600 leader_election.cc:288] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:33.905395 4605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:33.907405 4600 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:33.907356 4605 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [term 1 LEADER]: Becoming Leader. State: Replica: cb778469275943bc8da301a820439caf, State: Running, Role: LEADER
I20260926 23:21:33.908036 4605 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb778469275943bc8da301a820439caf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } }
I20260926 23:21:33.909178 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:33.913508 4606 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb778469275943bc8da301a820439caf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb778469275943bc8da301a820439caf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } } }
I20260926 23:21:33.913946 4606 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:33.913481 4607 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb778469275943bc8da301a820439caf. Latest consensus state: current_term: 1 leader_uuid: "cb778469275943bc8da301a820439caf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb778469275943bc8da301a820439caf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38017 } } }
I20260926 23:21:33.914577 4607 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:33.916673 4615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:33.921919 4615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:33.938652 4615 catalog_manager.cc:1347] Generated new cluster ID: 42b4d24029564850b134f6a2ec2776a1
I20260926 23:21:33.938783 4615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:33.954253 4615 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:33.955703 4615 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:33.956990 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:33.939371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49485 (local address 127.0.0.1:38017)
0926 23:21:33.939537 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:33.939543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:33.940753 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 23:21:33.940930 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:33.940935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:33.941005 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:33.941106 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:33.943001 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:33.944107 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:33.945853 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:33.946588 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:33.946718 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:33.949596 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:33.949603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:33.949609 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:33.949648 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:33.953574 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:33.954561 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:33.954567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:33.954570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:33.954740 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:33.955257 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:33.955262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:33.955265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:33.955635 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:33.955931 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:33.956387 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 23:21:33.956591 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260926 23:21:33.970867 4615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb778469275943bc8da301a820439caf: Generated new TSK 0
I20260926 23:21:33.971889 4615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:33.977545 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4533
I20260926 23:21:33.979388 4595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:34.070336 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 4533
W20260926 23:21:34.087414 4518 connection.cc:535] client connection to 127.0.0.1:38017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38017): Connection reset by peer (error 104)
2026-09-26T23:21:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:34 dist-test-slave-vcv1 krb5kdc[4632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:34 dist-test-slave-vcv1 krb5kdc[4632](info): set up 2 sockets
Sep 26 23:21:34 dist-test-slave-vcv1 krb5kdc[4632](info): commencing operation
krb5kdc: starting...
W20260926 23:21:35.169265 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:35 dist-test-slave-vcv1 krb5kdc[4632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:35.411764 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:35.729317 4645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:35.729588 4645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:35.729801 4645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:35.736964 4645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:35.737090 4645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:35.737174 4645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:35.737282 4645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:35.737360 4645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:35.750893 4645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:35.752213 4645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:35.753497 4645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:35.768620 4650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:35.768676 4651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:35.768638 4653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:35.769423 4645 server_base.cc:1034] running on GCE node
I20260926 23:21:35.770099 4645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:35.771345 4645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:35.772579 4645 hybrid_clock.cc:648] HybridClock initialized: now 1790464895772523 us; error 47 us; skew 500 ppm
I20260926 23:21:35.772979 4645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:35 dist-test-slave-vcv1 krb5kdc[4632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:35.776721 4645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:35.778083 4645 webserver.cc:460] Webserver started at http://127.0.0.1:38487/ using document root <none> and password file <none>
I20260926 23:21:35.778745 4645 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:35.778891 4645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:35.779196 4645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:35.782192 4645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "2fd07c4103244d1d9622dbd00e5bd6d9"
format_stamp: "Formatted at 2026-09-26 23:21:35 on dist-test-slave-vcv1"
server_key: "c047999e6e682adb7b513dce36c10e8e"
server_key_iv: "e6458436a7772446ee5665708145dbe7"
server_key_version: "encryptionkey@0"
I20260926 23:21:35.782939 4645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "2fd07c4103244d1d9622dbd00e5bd6d9"
format_stamp: "Formatted at 2026-09-26 23:21:35 on dist-test-slave-vcv1"
server_key: "c047999e6e682adb7b513dce36c10e8e"
server_key_iv: "e6458436a7772446ee5665708145dbe7"
server_key_version: "encryptionkey@0"
I20260926 23:21:35.787940 4645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:21:35.791745 4660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:35.792999 4645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:35.793188 4645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "2fd07c4103244d1d9622dbd00e5bd6d9"
format_stamp: "Formatted at 2026-09-26 23:21:35 on dist-test-slave-vcv1"
server_key: "c047999e6e682adb7b513dce36c10e8e"
server_key_iv: "e6458436a7772446ee5665708145dbe7"
server_key_version: "encryptionkey@0"
I20260926 23:21:35.793378 4645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:35.820195 4645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:35.824034 4645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:35.824349 4645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:35.842531 4645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36253
I20260926 23:21:35.842561 4711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36253 every 8 connection(s)
I20260926 23:21:35.843885 4645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:35.847484 4712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:35.853345 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 4645
I20260926 23:21:35.853523 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:35.854053 16633 external_mini_cluster.cc:1428] Setting key ea6db3b4444200f1517b17e41ceb24a4
I20260926 23:21:35.858494 4712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9: Bootstrap starting.
I20260926 23:21:35.862339 4712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:35.863514 4712 log.cc:824] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:35.866802 4712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9: No bootstrap required, opened a new log
I20260926 23:21:35.867839 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:35.855821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34497 (local address 127.0.0.1:36253)
0926 23:21:35.856704 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:35.856715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:35.856756 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:21:35.858384 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:35.858403 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:35.858840 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:35.859158 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:35.859709 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:35.861085 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:35.863116 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:35.863839 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:35.864030 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:35.865256 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:35.865266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:35.865285 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:35.865965 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:35.866040 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:35.866093 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:35.866259 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:35.866465 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 23:21:35.866621 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":451,"threads_started":1}
I20260926 23:21:35.871577 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:35.875054 4712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd07c4103244d1d9622dbd00e5bd6d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } }
I20260926 23:21:35.875442 4712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:35.875559 4712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd07c4103244d1d9622dbd00e5bd6d9, State: Initialized, Role: FOLLOWER
I20260926 23:21:35.876295 4712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd07c4103244d1d9622dbd00e5bd6d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } }
I20260926 23:21:35.876566 4712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:35.876718 4712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:35.876912 4712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:35.879137 4712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd07c4103244d1d9622dbd00e5bd6d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } }
I20260926 23:21:35.879630 4712 leader_election.cc:302] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fd07c4103244d1d9622dbd00e5bd6d9; no voters:
I20260926 23:21:35.880373 4712 leader_election.cc:288] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:35.880488 4722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:35.881773 4722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [term 1 LEADER]: Becoming Leader. State: Replica: 2fd07c4103244d1d9622dbd00e5bd6d9, State: Running, Role: LEADER
I20260926 23:21:35.882365 4722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd07c4103244d1d9622dbd00e5bd6d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } }
I20260926 23:21:35.882917 4712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:35.882977 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:35.876046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48749 (local address 127.0.0.1:36253)
0926 23:21:35.876273 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:35.876284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:35.876693 (+ 409us) server_negotiation.cc:407] Connection header received
0926 23:21:35.876882 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:35.876887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:35.876961 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:35.877059 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:35.877566 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:35.878348 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:35.879739 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:35.880428 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:35.880571 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:35.882083 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:35.882091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:35.882097 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:35.882116 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:35.882161 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:35.882183 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:35.882318 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:35.882573 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 23:21:35.882630 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:21:35.886202 4723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fd07c4103244d1d9622dbd00e5bd6d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd07c4103244d1d9622dbd00e5bd6d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } } }
I20260926 23:21:35.887241 4724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fd07c4103244d1d9622dbd00e5bd6d9. Latest consensus state: current_term: 1 leader_uuid: "2fd07c4103244d1d9622dbd00e5bd6d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd07c4103244d1d9622dbd00e5bd6d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } } }
I20260926 23:21:35.888621 4724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:35.888701 4723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:35.900928 4730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:35.908524 4730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:35.922621 4730 catalog_manager.cc:1347] Generated new cluster ID: f54af58401d54cba8953015398a20ae9
I20260926 23:21:35.922744 4730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:35.938982 4730 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:35.940438 4730 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:35.947465 4730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fd07c4103244d1d9622dbd00e5bd6d9: Generated new TSK 0
I20260926 23:21:35.948318 4730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:35.953634 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4645
W20260926 23:21:35.955207 4707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34497
W20260926 23:21:35.956041 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4645: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:21:35.956249 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 4645
W20260926 23:21:35.972963 4630 connection.cc:535] client connection to 127.0.0.1:36253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36253): Connection reset by peer (error 104)
2026-09-26T23:21:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:36 dist-test-slave-vcv1 krb5kdc[4743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:36 dist-test-slave-vcv1 krb5kdc[4743](info): set up 2 sockets
Sep 26 23:21:36 dist-test-slave-vcv1 krb5kdc[4743](info): commencing operation
krb5kdc: starting...
W20260926 23:21:37.061328 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:37 dist-test-slave-vcv1 krb5kdc[4743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:37.299556 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:37.619448 4756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:37.619731 4756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:37.619903 4756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:37.627174 4756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:37.627274 4756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:37.627338 4756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:37.627391 4756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:37.627478 4756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:37.637163 4756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:37.638420 4756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:37.639618 4756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:37.651052 4764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:37.651132 4762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:37.651297 4761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:37.651952 4756 server_base.cc:1034] running on GCE node
I20260926 23:21:37.652709 4756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:37.654024 4756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:37.655244 4756 hybrid_clock.cc:648] HybridClock initialized: now 1790464897655179 us; error 52 us; skew 500 ppm
I20260926 23:21:37.655602 4756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:37 dist-test-slave-vcv1 krb5kdc[4743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:37.663575 4756 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:37.664983 4756 webserver.cc:460] Webserver started at http://127.0.0.1:45223/ using document root <none> and password file <none>
I20260926 23:21:37.665676 4756 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:37.665820 4756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:37.666116 4756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:37.669000 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "53560a905e1a484391a5108d2465fd54"
format_stamp: "Formatted at 2026-09-26 23:21:37 on dist-test-slave-vcv1"
server_key: "156e1d1b33147a4438a87fc957c9b112"
server_key_iv: "177c277187b29ef5af94609a4000af8c"
server_key_version: "encryptionkey@0"
I20260926 23:21:37.669739 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "53560a905e1a484391a5108d2465fd54"
format_stamp: "Formatted at 2026-09-26 23:21:37 on dist-test-slave-vcv1"
server_key: "156e1d1b33147a4438a87fc957c9b112"
server_key_iv: "177c277187b29ef5af94609a4000af8c"
server_key_version: "encryptionkey@0"
I20260926 23:21:37.674964 4756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:21:37.678284 4771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:37.679445 4756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 23:21:37.679621 4756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "53560a905e1a484391a5108d2465fd54"
format_stamp: "Formatted at 2026-09-26 23:21:37 on dist-test-slave-vcv1"
server_key: "156e1d1b33147a4438a87fc957c9b112"
server_key_iv: "177c277187b29ef5af94609a4000af8c"
server_key_version: "encryptionkey@0"
I20260926 23:21:37.679803 4756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:37.685282 4756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:37.689236 4756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:37.689549 4756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:37.708228 4756 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43581
I20260926 23:21:37.708274 4822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43581 every 8 connection(s)
I20260926 23:21:37.709658 4756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:37.712752 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 4756
I20260926 23:21:37.712956 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:37.713418 16633 external_mini_cluster.cc:1428] Setting key 3f443731193e506e128255e37de39b38
I20260926 23:21:37.713466 4823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:37.721448 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:37.715061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52671 (local address 127.0.0.1:43581)
0926 23:21:37.715831 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:37.715842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:37.715884 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:21:37.718183 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:37.718205 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:37.718523 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:37.718825 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:37.719888 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:37.719894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:37.719905 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:37.719925 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:37.719982 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:37.720004 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:37.720235 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:37.720332 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 23:21:37.720546 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":337,"threads_started":1}
I20260926 23:21:37.725112 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:37.727769 4823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54: Bootstrap starting.
I20260926 23:21:37.731801 4823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:37.732658 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:37.730295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38381 (local address 127.0.0.1:43581)
0926 23:21:37.730484 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:37.730489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:37.730752 (+ 263us) server_negotiation.cc:407] Connection header received
0926 23:21:37.730923 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:37.730927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:37.730969 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:37.731041 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:37.732001 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:37.732006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:37.732008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:37.732023 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:37.732058 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:37.732071 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:37.732228 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:37.732336 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 23:21:37.732378 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:21:37.733170 4823 log.cc:824] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:37.735993 4823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54: No bootstrap required, opened a new log
I20260926 23:21:37.746258 4823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53560a905e1a484391a5108d2465fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } }
I20260926 23:21:37.746737 4823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:37.746845 4823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53560a905e1a484391a5108d2465fd54, State: Initialized, Role: FOLLOWER
I20260926 23:21:37.747475 4823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53560a905e1a484391a5108d2465fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } }
I20260926 23:21:37.747800 4823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:37.747932 4823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:37.748106 4823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:37.751523 4823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53560a905e1a484391a5108d2465fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } }
I20260926 23:21:37.752135 4823 leader_election.cc:302] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53560a905e1a484391a5108d2465fd54; no voters:
I20260926 23:21:37.752863 4823 leader_election.cc:288] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:37.752934 4833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:37.755349 4823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:37.756074 4833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [term 1 LEADER]: Becoming Leader. State: Replica: 53560a905e1a484391a5108d2465fd54, State: Running, Role: LEADER
I20260926 23:21:37.756649 4833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53560a905e1a484391a5108d2465fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } }
I20260926 23:21:37.760470 4835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53560a905e1a484391a5108d2465fd54. Latest consensus state: current_term: 1 leader_uuid: "53560a905e1a484391a5108d2465fd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53560a905e1a484391a5108d2465fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } } }
I20260926 23:21:37.760946 4835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:37.761475 4834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53560a905e1a484391a5108d2465fd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53560a905e1a484391a5108d2465fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } } }
I20260926 23:21:37.761785 4834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:37.764318 4845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:37.769553 4845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:37.779206 4845 catalog_manager.cc:1347] Generated new cluster ID: 28d24e1c2a3842e28f90979d3a9334dc
I20260926 23:21:37.779304 4845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:37.791795 4845 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:37.792272 4845 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:37.803869 4845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53560a905e1a484391a5108d2465fd54: Generated new TSK 0
I20260926 23:21:37.804822 4845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:37.817989 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4756
W20260926 23:21:37.819033 4818 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52671
W20260926 23:21:37.819757 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4756: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:21:37.819918 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 4756
2026-09-26T23:21:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1863 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:37 dist-test-slave-vcv1 krb5kdc[4854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:37 dist-test-slave-vcv1 krb5kdc[4854](info): set up 2 sockets
Sep 26 23:21:37 dist-test-slave-vcv1 krb5kdc[4854](info): commencing operation
krb5kdc: starting...
W20260926 23:21:38.917475 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:39 dist-test-slave-vcv1 krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:39.163826 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:39.479492 4867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:39.479758 4867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:39.479969 4867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:39.486914 4867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:39.487046 4867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:39.487129 4867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:39.487241 4867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:39.487326 4867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:39.496713 4867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:39.497944 4867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:39.499225 4867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:39.511147 4872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:39.511340 4873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:39.511461 4875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:39.512420 4867 server_base.cc:1034] running on GCE node
I20260926 23:21:39.513135 4867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:39.514536 4867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:39.515758 4867 hybrid_clock.cc:648] HybridClock initialized: now 1790464899515723 us; error 33 us; skew 500 ppm
I20260926 23:21:39.516099 4867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:39 dist-test-slave-vcv1 krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:39.519106 4867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:39.520548 4867 webserver.cc:460] Webserver started at http://127.0.0.1:35889/ using document root <none> and password file <none>
I20260926 23:21:39.521194 4867 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:39.521338 4867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:39.521642 4867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:39.524583 4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "7316273802854e2bb37848e5158e9f05"
format_stamp: "Formatted at 2026-09-26 23:21:39 on dist-test-slave-vcv1"
server_key: "f20a42068e74b8fdcd63771fd9084126"
server_key_iv: "fdda38eb000be3a0fed6698a64455954"
server_key_version: "encryptionkey@0"
I20260926 23:21:39.525353 4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "7316273802854e2bb37848e5158e9f05"
format_stamp: "Formatted at 2026-09-26 23:21:39 on dist-test-slave-vcv1"
server_key: "f20a42068e74b8fdcd63771fd9084126"
server_key_iv: "fdda38eb000be3a0fed6698a64455954"
server_key_version: "encryptionkey@0"
I20260926 23:21:39.530314 4867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:21:39.533634 4882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:39.534940 4867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:21:39.535121 4867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "7316273802854e2bb37848e5158e9f05"
format_stamp: "Formatted at 2026-09-26 23:21:39 on dist-test-slave-vcv1"
server_key: "f20a42068e74b8fdcd63771fd9084126"
server_key_iv: "fdda38eb000be3a0fed6698a64455954"
server_key_version: "encryptionkey@0"
I20260926 23:21:39.535315 4867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:39.560915 4867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:39.564781 4867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:39.565059 4867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:39.583343 4867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41883
I20260926 23:21:39.583379 4933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41883 every 8 connection(s)
I20260926 23:21:39.584750 4867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:39.586946 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 4867
I20260926 23:21:39.587131 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:39.587572 16633 external_mini_cluster.cc:1428] Setting key d820682ca45e92d7e7495d35f3226b0c
I20260926 23:21:39.588614 4934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:21:39 dist-test-slave-vcv1 krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:21:39.602051 4934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05: Bootstrap starting.
I20260926 23:21:39.606189 4934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:39.607446 4934 log.cc:824] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:39.610615 4934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05: No bootstrap required, opened a new log
I20260926 23:21:39.610879 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:39.589346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37951 (local address 127.0.0.1:41883)
0926 23:21:39.590166 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:39.590180 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:39.590228 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:21:39.591991 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:39.592016 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:39.592549 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:39.592977 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:39.594183 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:39.596257 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:39.597478 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:39.598467 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:39.598666 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:39.602581 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:39.602591 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:39.602607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:39.602647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:39.606626 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:39.607284 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:39.607293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:39.607299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:39.607508 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:39.607921 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:39.607929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:39.607935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:39.608584 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:39.608863 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:39.609235 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 23:21:39.609645 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":358,"threads_started":1}
I20260926 23:21:39.614401 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:39.618498 4934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316273802854e2bb37848e5158e9f05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41883 } }
I20260926 23:21:39.618914 4934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:39.619028 4934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7316273802854e2bb37848e5158e9f05, State: Initialized, Role: FOLLOWER
I20260926 23:21:39.619648 4934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316273802854e2bb37848e5158e9f05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41883 } }
I20260926 23:21:39.619939 4934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:39.620095 4934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:39.620364 4934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:39.623248 4934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316273802854e2bb37848e5158e9f05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41883 } }
I20260926 23:21:39.623795 4934 leader_election.cc:302] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7316273802854e2bb37848e5158e9f05; no voters:
I20260926 23:21:39.624557 4934 leader_election.cc:288] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:39.624684 4944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:39.625700 4944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [term 1 LEADER]: Becoming Leader. State: Replica: 7316273802854e2bb37848e5158e9f05, State: Running, Role: LEADER
I20260926 23:21:39.626457 4944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316273802854e2bb37848e5158e9f05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41883 } }
I20260926 23:21:39.628727 4944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7316273802854e2bb37848e5158e9f05. Latest consensus state: current_term: 1 leader_uuid: "7316273802854e2bb37848e5158e9f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316273802854e2bb37848e5158e9f05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41883 } } }
I20260926 23:21:39.630594 4944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:39.630142 4946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7316273802854e2bb37848e5158e9f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316273802854e2bb37848e5158e9f05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41883 } } }
I20260926 23:21:39.631065 4946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:39.634119 4934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:39.634317 4948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:39.634833 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:39.618504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55895 (local address 127.0.0.1:41883)
0926 23:21:39.618711 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:39.618720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:39.619192 (+ 472us) server_negotiation.cc:407] Connection header received
0926 23:21:39.619343 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:39.619350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:39.619443 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:39.619527 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:39.621383 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:39.622109 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:39.624048 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:39.624617 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:39.624809 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:39.627415 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:39.627421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:39.627425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:39.627463 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:39.631970 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:39.632792 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:39.632799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:39.632804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:39.632970 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:39.633348 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:39.633353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:39.633356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:39.633639 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:39.633934 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:39.634244 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 23:21:39.634465 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:21:39.640494 4948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:39.655622 4948 catalog_manager.cc:1347] Generated new cluster ID: 4dbc24fcc6a946bd9cb51b2eb351851d
I20260926 23:21:39.655752 4948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:39.675434 4948 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:39.677395 4948 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:39.687093 4948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7316273802854e2bb37848e5158e9f05: Generated new TSK 0
I20260926 23:21:39.687986 4948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:39.697721 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4867
I20260926 23:21:39.699147 4929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:39.794787 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 4867
2026-09-26T23:21:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:39 dist-test-slave-vcv1 krb5kdc[4966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:39 dist-test-slave-vcv1 krb5kdc[4966](info): set up 2 sockets
Sep 26 23:21:39 dist-test-slave-vcv1 krb5kdc[4966](info): commencing operation
krb5kdc: starting...
W20260926 23:21:40.909468 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:41 dist-test-slave-vcv1 krb5kdc[4966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:41.149592 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:41.473744 4979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:41.474025 4979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:41.474220 4979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:41.481410 4979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:41.481518 4979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:41.481564 4979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:41.481662 4979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:41.481726 4979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:41.491350 4979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:41.492622 4979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:41.493866 4979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:41.505401 4985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:41.505443 4984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:41.505482 4987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:41.506486 4979 server_base.cc:1034] running on GCE node
I20260926 23:21:41.507230 4979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:41.508626 4979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:41.509860 4979 hybrid_clock.cc:648] HybridClock initialized: now 1790464901509826 us; error 47 us; skew 500 ppm
I20260926 23:21:41.510191 4979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:41 dist-test-slave-vcv1 krb5kdc[4966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:41.513636 4979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:41.515055 4979 webserver.cc:460] Webserver started at http://127.0.0.1:40519/ using document root <none> and password file <none>
I20260926 23:21:41.515748 4979 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:41.515898 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:41.516273 4979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:41.519249 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "99a8e32455284233b9300c43eecdad6a"
format_stamp: "Formatted at 2026-09-26 23:21:41 on dist-test-slave-vcv1"
server_key: "5231dbf63c2a19779db22b998055eb19"
server_key_iv: "0bc4e7230f5b921a3ef005aac98f216d"
server_key_version: "encryptionkey@0"
I20260926 23:21:41.520020 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "99a8e32455284233b9300c43eecdad6a"
format_stamp: "Formatted at 2026-09-26 23:21:41 on dist-test-slave-vcv1"
server_key: "5231dbf63c2a19779db22b998055eb19"
server_key_iv: "0bc4e7230f5b921a3ef005aac98f216d"
server_key_version: "encryptionkey@0"
I20260926 23:21:41.525156 4979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:21:41.528452 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:41.529755 4979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:21:41.529947 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "99a8e32455284233b9300c43eecdad6a"
format_stamp: "Formatted at 2026-09-26 23:21:41 on dist-test-slave-vcv1"
server_key: "5231dbf63c2a19779db22b998055eb19"
server_key_iv: "0bc4e7230f5b921a3ef005aac98f216d"
server_key_version: "encryptionkey@0"
I20260926 23:21:41.530157 4979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:41.541532 4979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:41.545746 4979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:41.546033 4979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:41.563364 4979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43679
I20260926 23:21:41.563395 5045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43679 every 8 connection(s)
I20260926 23:21:41.564782 4979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:41.568465 5046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:41.573371 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 4979
I20260926 23:21:41.573547 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:41.574018 16633 external_mini_cluster.cc:1428] Setting key 781bf1dc1600335db79801b3aa7fc133
I20260926 23:21:41.580471 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a: Bootstrap starting.
I20260926 23:21:41.584048 5046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:41.585399 5046 log.cc:824] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:41.588451 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a: No bootstrap required, opened a new log
I20260926 23:21:41.588723 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:41.575871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55495 (local address 127.0.0.1:43679)
0926 23:21:41.576761 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:41.576775 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:41.576825 (+ 50us) server_negotiation.cc:407] Connection header received
0926 23:21:41.579257 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:41.579284 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:41.579777 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:41.580217 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:41.580728 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:41.582738 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:41.584061 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:41.585117 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:41.585309 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:41.586543 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:41.586552 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:41.586568 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:41.587059 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:41.587132 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:41.587155 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:41.587276 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:41.587485 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 23:21:41.587720 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":400,"threads_started":1}
I20260926 23:21:41.592175 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:41.596539 5046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a8e32455284233b9300c43eecdad6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260926 23:21:41.596982 5046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:41.597102 5046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a8e32455284233b9300c43eecdad6a, State: Initialized, Role: FOLLOWER
I20260926 23:21:41.597679 5046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a8e32455284233b9300c43eecdad6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260926 23:21:41.597971 5046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:41.598109 5046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:41.598291 5046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:41.600867 5046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a8e32455284233b9300c43eecdad6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260926 23:21:41.601388 5046 leader_election.cc:302] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99a8e32455284233b9300c43eecdad6a; no voters:
I20260926 23:21:41.601998 5046 leader_election.cc:288] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:41.602094 5056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:41.603435 5056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [term 1 LEADER]: Becoming Leader. State: Replica: 99a8e32455284233b9300c43eecdad6a, State: Running, Role: LEADER
I20260926 23:21:41.603556 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:41.596281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40447 (local address 127.0.0.1:43679)
0926 23:21:41.596457 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:41.596465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:41.596824 (+ 359us) server_negotiation.cc:407] Connection header received
0926 23:21:41.597067 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:41.597072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:41.597156 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:41.597266 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:41.597860 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:41.598944 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:41.600266 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:41.600932 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:41.601067 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:41.602237 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:41.602244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:41.602252 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:41.602281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:41.602343 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:41.602382 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:41.602974 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:41.603033 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 23:21:41.603110 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 23:21:41.604223 5056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a8e32455284233b9300c43eecdad6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260926 23:21:41.604452 5046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:41.607452 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99a8e32455284233b9300c43eecdad6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a8e32455284233b9300c43eecdad6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } } }
I20260926 23:21:41.608027 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:41.607431 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99a8e32455284233b9300c43eecdad6a. Latest consensus state: current_term: 1 leader_uuid: "99a8e32455284233b9300c43eecdad6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a8e32455284233b9300c43eecdad6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } } }
I20260926 23:21:41.608597 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a [sys.catalog]: This master's current role is: LEADER
W20260926 23:21:41.617792 5072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:21:41.617933 5072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:21:41.618084 5064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:41.619508 5064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:41.627688 5064 catalog_manager.cc:1347] Generated new cluster ID: 56029f14ab8643ac82827cdd8490f55d
I20260926 23:21:41.627796 5064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:41.655869 5064 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:41.657167 5064 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:41.663615 5064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99a8e32455284233b9300c43eecdad6a: Generated new TSK 0
I20260926 23:21:41.664646 5064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:41.670830 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4979
W20260926 23:21:41.671906 5041 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55495
W20260926 23:21:41.672577 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 4979: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:21:41.672729 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 4979
2026-09-26T23:21:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:41 dist-test-slave-vcv1 krb5kdc[5077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:41 dist-test-slave-vcv1 krb5kdc[5077](info): set up 2 sockets
Sep 26 23:21:41 dist-test-slave-vcv1 krb5kdc[5077](info): commencing operation
krb5kdc: starting...
W20260926 23:21:42.797595 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:42 dist-test-slave-vcv1 krb5kdc[5077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:43.043071 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:43.366557 5090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:43.366817 5090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:43.366977 5090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:43.373982 5090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:43.374075 5090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:43.374126 5090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:43.374217 5090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:43.374279 5090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:43.383560 5090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:43.384738 5090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:43.385995 5090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:43.397405 5095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:43.397464 5098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:43.397500 5096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:43.398458 5090 server_base.cc:1034] running on GCE node
I20260926 23:21:43.399147 5090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:43.400523 5090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:43.401747 5090 hybrid_clock.cc:648] HybridClock initialized: now 1790464903401705 us; error 38 us; skew 500 ppm
I20260926 23:21:43.402103 5090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:43 dist-test-slave-vcv1 krb5kdc[5077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:43.410913 5090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:43.412367 5090 webserver.cc:460] Webserver started at http://127.0.0.1:45571/ using document root <none> and password file <none>
I20260926 23:21:43.413014 5090 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:43.413167 5090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:43.413470 5090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:43.416499 5090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "4581b574076b45c68dc329fda5b2b89c"
format_stamp: "Formatted at 2026-09-26 23:21:43 on dist-test-slave-vcv1"
server_key: "517bd0502676656a698fd77f8f6fedbc"
server_key_iv: "bc2582f6fa89cc27e631d0621ea91de5"
server_key_version: "encryptionkey@0"
I20260926 23:21:43.417264 5090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "4581b574076b45c68dc329fda5b2b89c"
format_stamp: "Formatted at 2026-09-26 23:21:43 on dist-test-slave-vcv1"
server_key: "517bd0502676656a698fd77f8f6fedbc"
server_key_iv: "bc2582f6fa89cc27e631d0621ea91de5"
server_key_version: "encryptionkey@0"
I20260926 23:21:43.422096 5090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:21:43.425300 5105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:43.426474 5090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 23:21:43.426651 5090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "4581b574076b45c68dc329fda5b2b89c"
format_stamp: "Formatted at 2026-09-26 23:21:43 on dist-test-slave-vcv1"
server_key: "517bd0502676656a698fd77f8f6fedbc"
server_key_iv: "bc2582f6fa89cc27e631d0621ea91de5"
server_key_version: "encryptionkey@0"
I20260926 23:21:43.426836 5090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:43.432360 5090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:43.436538 5090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:43.436841 5090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:43.455238 5090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41351
I20260926 23:21:43.455287 5156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41351 every 8 connection(s)
I20260926 23:21:43.456655 5090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:43.460471 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:43.466162 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 5090
I20260926 23:21:43.466341 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:43.466760 16633 external_mini_cluster.cc:1428] Setting key 7b51fa7a0c5c4f4043a5fd55a545c796
I20260926 23:21:43.470887 5157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c: Bootstrap starting.
I20260926 23:21:43.474225 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:43.468533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54363 (local address 127.0.0.1:41351)
0926 23:21:43.469346 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:43.469356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:43.469394 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:21:43.471035 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:43.471065 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:43.471379 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:43.471695 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:43.472789 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:43.472795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:43.472806 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:43.472820 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:43.472876 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:43.472895 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:43.473062 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:43.473186 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 23:21:43.473380 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":379,"threads_started":1}
I20260926 23:21:43.474460 5157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:43.475744 5157 log.cc:824] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:43.478111 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:43.478847 5157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c: No bootstrap required, opened a new log
I20260926 23:21:43.486565 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:43.483176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35099 (local address 127.0.0.1:41351)
0926 23:21:43.483376 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:43.483383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:43.484670 (+ 1287us) server_negotiation.cc:407] Connection header received
0926 23:21:43.484812 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:43.484817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:43.484884 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:43.485012 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:43.485871 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:43.485877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:43.485882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:43.485898 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:43.485941 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:43.485975 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:43.486107 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:43.486213 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 23:21:43.486268 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:21:43.488220 5157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581b574076b45c68dc329fda5b2b89c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260926 23:21:43.488585 5157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:43.488699 5157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4581b574076b45c68dc329fda5b2b89c, State: Initialized, Role: FOLLOWER
I20260926 23:21:43.489293 5157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581b574076b45c68dc329fda5b2b89c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260926 23:21:43.489543 5157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:43.489681 5157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:43.489856 5157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:43.492122 5157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581b574076b45c68dc329fda5b2b89c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260926 23:21:43.492655 5157 leader_election.cc:302] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4581b574076b45c68dc329fda5b2b89c; no voters:
I20260926 23:21:43.493211 5157 leader_election.cc:288] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:43.493350 5167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:43.494000 5167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [term 1 LEADER]: Becoming Leader. State: Replica: 4581b574076b45c68dc329fda5b2b89c, State: Running, Role: LEADER
I20260926 23:21:43.494536 5167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581b574076b45c68dc329fda5b2b89c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260926 23:21:43.495159 5157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:43.499030 5169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4581b574076b45c68dc329fda5b2b89c. Latest consensus state: current_term: 1 leader_uuid: "4581b574076b45c68dc329fda5b2b89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581b574076b45c68dc329fda5b2b89c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260926 23:21:43.498852 5168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4581b574076b45c68dc329fda5b2b89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4581b574076b45c68dc329fda5b2b89c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260926 23:21:43.499439 5169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:43.499457 5168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:43.500388 5176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:43.511772 5176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:43.521977 5176 catalog_manager.cc:1347] Generated new cluster ID: 344a4f791a9648cca6d57f6a536ad0cf
I20260926 23:21:43.522166 5176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:43.547063 5176 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:43.547487 5176 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:43.552783 5176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4581b574076b45c68dc329fda5b2b89c: Generated new TSK 0
I20260926 23:21:43.553560 5176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:43.561013 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5090
W20260926 23:21:43.562125 5152 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54363
W20260926 23:21:43.562803 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5090: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:21:43.562949 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 5090
2026-09-26T23:21:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:43 dist-test-slave-vcv1 krb5kdc[5188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:43 dist-test-slave-vcv1 krb5kdc[5188](info): set up 2 sockets
Sep 26 23:21:43 dist-test-slave-vcv1 krb5kdc[5188](info): commencing operation
krb5kdc: starting...
W20260926 23:21:44.657569 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:44 dist-test-slave-vcv1 krb5kdc[5188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:44.897507 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:45.217417 5201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:45.217694 5201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:45.217877 5201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:45.224984 5201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:45.225086 5201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:45.225199 5201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:45.225306 5201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:45.225385 5201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:45.234917 5201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:45.236189 5201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:45.237469 5201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:45.249145 5206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:45.249207 5209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:45.249274 5207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:45.249969 5201 server_base.cc:1034] running on GCE node
I20260926 23:21:45.250666 5201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:45.251910 5201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:45.253118 5201 hybrid_clock.cc:648] HybridClock initialized: now 1790464905253081 us; error 33 us; skew 500 ppm
I20260926 23:21:45.253487 5201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:45 dist-test-slave-vcv1 krb5kdc[5188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:45.256667 5201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:45.258072 5201 webserver.cc:460] Webserver started at http://127.0.0.1:41607/ using document root <none> and password file <none>
I20260926 23:21:45.258723 5201 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:45.258870 5201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:45.259172 5201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:45.262087 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "7b24e6017a714503b201dc243777df78"
format_stamp: "Formatted at 2026-09-26 23:21:45 on dist-test-slave-vcv1"
server_key: "19bc09a6042b43c8b6b5399fa8300747"
server_key_iv: "fd73e96d38c6c226abbece864823e416"
server_key_version: "encryptionkey@0"
I20260926 23:21:45.262867 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "7b24e6017a714503b201dc243777df78"
format_stamp: "Formatted at 2026-09-26 23:21:45 on dist-test-slave-vcv1"
server_key: "19bc09a6042b43c8b6b5399fa8300747"
server_key_iv: "fd73e96d38c6c226abbece864823e416"
server_key_version: "encryptionkey@0"
I20260926 23:21:45.268263 5201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 23:21:45.271492 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:45.272763 5201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:45.272940 5201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "7b24e6017a714503b201dc243777df78"
format_stamp: "Formatted at 2026-09-26 23:21:45 on dist-test-slave-vcv1"
server_key: "19bc09a6042b43c8b6b5399fa8300747"
server_key_iv: "fd73e96d38c6c226abbece864823e416"
server_key_version: "encryptionkey@0"
I20260926 23:21:45.273130 5201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:45.288259 5201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:45.292007 5201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:45.292337 5201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:45.310343 5201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41969
I20260926 23:21:45.310377 5267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41969 every 8 connection(s)
I20260926 23:21:45.311661 5201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:45.315124 5268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:45.319903 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 5201
I20260926 23:21:45.320091 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:45.320550 16633 external_mini_cluster.cc:1428] Setting key 3396238c2e0169e29c9f13b5821a2d6d
I20260926 23:21:45.326931 5268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78: Bootstrap starting.
I20260926 23:21:45.330581 5268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:45.331703 5268 log.cc:824] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:45.334368 5268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78: No bootstrap required, opened a new log
Sep 26 23:21:45 dist-test-slave-vcv1 krb5kdc[5188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:21:45.341742 5268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24e6017a714503b201dc243777df78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260926 23:21:45.342115 5268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:45.342234 5268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b24e6017a714503b201dc243777df78, State: Initialized, Role: FOLLOWER
I20260926 23:21:45.342923 5268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24e6017a714503b201dc243777df78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260926 23:21:45.343217 5268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:45.343390 5268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:45.343571 5268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:45.344081 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:45.322404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:37891 (local address 127.0.0.1:41969)
0926 23:21:45.323179 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:45.323193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:45.323246 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:21:45.325672 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:45.325698 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:45.326172 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:45.326618 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:45.328004 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:45.330044 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:45.331457 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:45.332534 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:45.332761 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:45.336855 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:45.336862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:45.336874 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:45.336911 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:45.339922 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:45.340718 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:45.340725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:45.340730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:45.340885 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:45.341355 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:45.341365 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:45.341367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:45.341843 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:45.342168 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:45.342669 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 23:21:45.342966 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":313,"threads_started":1}
I20260926 23:21:45.346535 5268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24e6017a714503b201dc243777df78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260926 23:21:45.347054 5268 leader_election.cc:302] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b24e6017a714503b201dc243777df78; no voters:
I20260926 23:21:45.347707 5268 leader_election.cc:288] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:45.347811 5273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:45.348073 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:45.349606 5268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:45.354790 5273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [term 1 LEADER]: Becoming Leader. State: Replica: 7b24e6017a714503b201dc243777df78, State: Running, Role: LEADER
I20260926 23:21:45.355387 5273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24e6017a714503b201dc243777df78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260926 23:21:45.358379 5275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b24e6017a714503b201dc243777df78. Latest consensus state: current_term: 1 leader_uuid: "7b24e6017a714503b201dc243777df78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24e6017a714503b201dc243777df78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } } }
I20260926 23:21:45.358767 5275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:45.358945 5274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b24e6017a714503b201dc243777df78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24e6017a714503b201dc243777df78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } } }
I20260926 23:21:45.359256 5274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:45.375804 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:45.356645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:36491 (local address 127.0.0.1:41969)
0926 23:21:45.356816 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:45.356823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:45.357172 (+ 349us) server_negotiation.cc:407] Connection header received
0926 23:21:45.357366 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:45.357372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:45.357466 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:45.357568 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:45.359783 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:45.361336 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:45.363794 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:45.364758 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:45.364901 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:45.369111 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:45.369121 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:45.369128 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:45.369178 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:45.373042 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:45.373745 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:45.373752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:45.373761 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:45.373874 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:45.374356 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:45.374359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:45.374361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:45.374658 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:45.374932 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:45.375316 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 23:21:45.375500 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 23:21:45.376012 5294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:21:45.376199 5294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:21:45.376335 5292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:45.378430 5292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:45.387168 5292 catalog_manager.cc:1347] Generated new cluster ID: 8df4c6db5b5f464cb59710b9dde44cae
I20260926 23:21:45.387284 5292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:45.405017 5292 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:45.406203 5292 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:45.414253 5292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b24e6017a714503b201dc243777df78: Generated new TSK 0
I20260926 23:21:45.414996 5292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:45.424211 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5201
I20260926 23:21:45.425340 5263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:45.515026 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 5201
W20260926 23:21:45.532128 5186 connection.cc:535] client connection to 127.0.0.1:41969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41969): Connection reset by peer (error 104)
2026-09-26T23:21:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:45 dist-test-slave-vcv1 krb5kdc[5300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:45 dist-test-slave-vcv1 krb5kdc[5300](info): set up 2 sockets
Sep 26 23:21:45 dist-test-slave-vcv1 krb5kdc[5300](info): commencing operation
krb5kdc: starting...
W20260926 23:21:46.609429 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:46 dist-test-slave-vcv1 krb5kdc[5300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:46.855309 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:47.173959 5313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:47.174228 5313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:47.174407 5313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:47.181257 5313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:47.181347 5313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:47.181481 5313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:47.181597 5313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:47.181679 5313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:47.190936 5313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:47.192149 5313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:47.193473 5313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:47.204990 5318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:47.205016 5321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:47.204949 5319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:47.205778 5313 server_base.cc:1034] running on GCE node
I20260926 23:21:47.206537 5313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:47.207811 5313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:47.209046 5313 hybrid_clock.cc:648] HybridClock initialized: now 1790464907208994 us; error 46 us; skew 500 ppm
I20260926 23:21:47.209390 5313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:47 dist-test-slave-vcv1 krb5kdc[5300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:47.212607 5313 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:47.213965 5313 webserver.cc:460] Webserver started at http://127.0.0.1:41441/ using document root <none> and password file <none>
I20260926 23:21:47.214538 5313 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:47.214636 5313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:47.214867 5313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:47.217680 5313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "3a5b584a9fcf4177add4d93666bfe5b7"
format_stamp: "Formatted at 2026-09-26 23:21:47 on dist-test-slave-vcv1"
server_key: "f67016f4a5db8dfc8162bbba3cd798b4"
server_key_iv: "2aad02b72ae72fc36a1a7926d3a108f6"
server_key_version: "encryptionkey@0"
I20260926 23:21:47.218365 5313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "3a5b584a9fcf4177add4d93666bfe5b7"
format_stamp: "Formatted at 2026-09-26 23:21:47 on dist-test-slave-vcv1"
server_key: "f67016f4a5db8dfc8162bbba3cd798b4"
server_key_iv: "2aad02b72ae72fc36a1a7926d3a108f6"
server_key_version: "encryptionkey@0"
I20260926 23:21:47.223088 5313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 23:21:47.226171 5328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:47.227370 5313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:21:47.227492 5313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "3a5b584a9fcf4177add4d93666bfe5b7"
format_stamp: "Formatted at 2026-09-26 23:21:47 on dist-test-slave-vcv1"
server_key: "f67016f4a5db8dfc8162bbba3cd798b4"
server_key_iv: "2aad02b72ae72fc36a1a7926d3a108f6"
server_key_version: "encryptionkey@0"
I20260926 23:21:47.227638 5313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:47.246595 5313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:47.250831 5313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:47.251067 5313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:47.269172 5313 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33131
I20260926 23:21:47.269207 5379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33131 every 8 connection(s)
I20260926 23:21:47.270495 5313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:47.274051 5380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:47.278373 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 5313
I20260926 23:21:47.278549 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:47.278987 16633 external_mini_cluster.cc:1428] Setting key dc5a3cde8ff1a7d6ab48919016fdb29e
I20260926 23:21:47.286263 5380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7: Bootstrap starting.
I20260926 23:21:47.289988 5380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:47.291219 5380 log.cc:824] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:47.294307 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:47.280814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:45273 (local address 127.0.0.1:33131)
0926 23:21:47.281574 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:47.281590 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:47.281653 (+ 63us) server_negotiation.cc:407] Connection header received
0926 23:21:47.284443 (+ 2790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:47.284471 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:47.284971 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:47.285388 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:47.285907 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:47.287868 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:47.289261 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:47.290335 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:47.290550 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:47.291635 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:47.291647 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:47.291668 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:47.292399 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:47.292503 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:47.292534 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:47.292708 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:47.292950 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 23:21:47.293105 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":301,"threads_started":1}
I20260926 23:21:47.294545 5380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7: No bootstrap required, opened a new log
I20260926 23:21:47.298189 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:47.302490 5380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b584a9fcf4177add4d93666bfe5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260926 23:21:47.302953 5380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:47.303076 5380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5b584a9fcf4177add4d93666bfe5b7, State: Initialized, Role: FOLLOWER
I20260926 23:21:47.303725 5380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b584a9fcf4177add4d93666bfe5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260926 23:21:47.304032 5380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:47.304265 5380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:47.304450 5380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:47.307526 5380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b584a9fcf4177add4d93666bfe5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260926 23:21:47.308099 5380 leader_election.cc:302] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a5b584a9fcf4177add4d93666bfe5b7; no voters:
I20260926 23:21:47.308840 5380 leader_election.cc:288] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:47.308970 5390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:47.309628 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:47.302388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:39337 (local address 127.0.0.1:33131)
0926 23:21:47.302590 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:47.302599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:47.303097 (+ 498us) server_negotiation.cc:407] Connection header received
0926 23:21:47.303291 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:47.303295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:47.303375 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:47.303491 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:47.304154 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:47.305035 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:47.306857 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:47.307293 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:47.307378 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:47.308673 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:47.308683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:47.308693 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:47.308716 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:47.308768 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:47.308798 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:47.308925 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:47.309183 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 23:21:47.309235 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:21:47.309810 5390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [term 1 LEADER]: Becoming Leader. State: Replica: 3a5b584a9fcf4177add4d93666bfe5b7, State: Running, Role: LEADER
I20260926 23:21:47.310374 5390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b584a9fcf4177add4d93666bfe5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260926 23:21:47.312593 5390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a5b584a9fcf4177add4d93666bfe5b7. Latest consensus state: current_term: 1 leader_uuid: "3a5b584a9fcf4177add4d93666bfe5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b584a9fcf4177add4d93666bfe5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } } }
I20260926 23:21:47.313014 5390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:47.313441 5392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a5b584a9fcf4177add4d93666bfe5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b584a9fcf4177add4d93666bfe5b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } } }
I20260926 23:21:47.313819 5392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:47.314373 5380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:47.319833 5394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:47.325089 5394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:47.333591 5394 catalog_manager.cc:1347] Generated new cluster ID: 513d8b2e22e54aaf8913aa0bba046773
I20260926 23:21:47.333705 5394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:47.348017 5394 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:47.349709 5394 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:47.358212 5394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a5b584a9fcf4177add4d93666bfe5b7: Generated new TSK 0
I20260926 23:21:47.359357 5394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:47.370555 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5313
W20260926 23:21:47.372022 5375 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.106:45273
W20260926 23:21:47.372828 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5313: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:21:47.372973 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 5313
W20260926 23:21:47.389276 5298 connection.cc:535] client connection to 127.0.0.1:33131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33131): Connection reset by peer (error 104)
2026-09-26T23:21:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:47 dist-test-slave-vcv1 krb5kdc[5411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:47 dist-test-slave-vcv1 krb5kdc[5411](info): set up 2 sockets
Sep 26 23:21:47 dist-test-slave-vcv1 krb5kdc[5411](info): commencing operation
krb5kdc: starting...
W20260926 23:21:48.465718 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:48 dist-test-slave-vcv1 krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:48.709026 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:49.037690 5424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:49.037971 5424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:49.038069 5424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:49.045142 5424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:49.045249 5424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:49.045295 5424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:49.045358 5424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:49.045401 5424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:49.055568 5424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:49.056774 5424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:49.057963 5424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:49.069952 5430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:49.069913 5429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:49.069991 5432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:49.070824 5424 server_base.cc:1034] running on GCE node
I20260926 23:21:49.071557 5424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:49.072999 5424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:49.074232 5424 hybrid_clock.cc:648] HybridClock initialized: now 1790464909074195 us; error 59 us; skew 500 ppm
I20260926 23:21:49.074587 5424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:49 dist-test-slave-vcv1 krb5kdc[5411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:49.082398 5424 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:49.083985 5424 webserver.cc:460] Webserver started at http://127.0.0.1:40451/ using document root <none> and password file <none>
I20260926 23:21:49.084682 5424 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:49.084784 5424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:49.085044 5424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:49.087891 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "27efe785ba7745638ef2aa16a2fd0e14"
format_stamp: "Formatted at 2026-09-26 23:21:49 on dist-test-slave-vcv1"
server_key: "cfe8f49a4e2475519e7876bffbf65eed"
server_key_iv: "353f189c10f4463ab1332379f2b59c5d"
server_key_version: "encryptionkey@0"
I20260926 23:21:49.088665 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "27efe785ba7745638ef2aa16a2fd0e14"
format_stamp: "Formatted at 2026-09-26 23:21:49 on dist-test-slave-vcv1"
server_key: "cfe8f49a4e2475519e7876bffbf65eed"
server_key_iv: "353f189c10f4463ab1332379f2b59c5d"
server_key_version: "encryptionkey@0"
I20260926 23:21:49.093657 5424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:21:49.097090 5439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:49.098368 5424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:49.098502 5424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "27efe785ba7745638ef2aa16a2fd0e14"
format_stamp: "Formatted at 2026-09-26 23:21:49 on dist-test-slave-vcv1"
server_key: "cfe8f49a4e2475519e7876bffbf65eed"
server_key_iv: "353f189c10f4463ab1332379f2b59c5d"
server_key_version: "encryptionkey@0"
I20260926 23:21:49.098668 5424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:49.104264 5424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:49.108096 5424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:49.108407 5424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:49.128686 5424 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44807
I20260926 23:21:49.128756 5490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44807 every 8 connection(s)
I20260926 23:21:49.130034 5424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:49.133533 5491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:49.134791 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 5424
I20260926 23:21:49.134982 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:49.135560 16633 external_mini_cluster.cc:1428] Setting key e5c2deb0640e5f7bb4525c95d1dc74c7
I20260926 23:21:49.143136 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:49.137394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:57119 (local address 127.0.0.1:44807)
0926 23:21:49.138178 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:49.138190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:49.138230 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:21:49.139855 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:49.139875 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:49.140277 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:49.140599 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:49.141638 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:49.141645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:49.141657 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:49.141671 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:49.141731 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:49.141750 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:49.141912 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:49.142033 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 23:21:49.142241 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":322,"threads_started":1}
I20260926 23:21:49.147169 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:49.147367 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14: Bootstrap starting.
I20260926 23:21:49.151130 5491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:49.152486 5491 log.cc:824] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:49.155084 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:49.152478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:36441 (local address 127.0.0.1:44807)
0926 23:21:49.152655 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:49.152661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:49.153203 (+ 542us) server_negotiation.cc:407] Connection header received
0926 23:21:49.153337 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:49.153348 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:49.153410 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:49.153533 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:49.154428 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:49.154443 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:49.154447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:49.154466 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:49.154514 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:49.154529 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:49.154644 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:49.154767 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 23:21:49.154824 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:21:49.155392 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14: No bootstrap required, opened a new log
I20260926 23:21:49.162562 5491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27efe785ba7745638ef2aa16a2fd0e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260926 23:21:49.162884 5491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:49.163048 5491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27efe785ba7745638ef2aa16a2fd0e14, State: Initialized, Role: FOLLOWER
I20260926 23:21:49.163590 5491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27efe785ba7745638ef2aa16a2fd0e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260926 23:21:49.163826 5491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:49.163975 5491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:49.164131 5491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:49.166314 5491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27efe785ba7745638ef2aa16a2fd0e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260926 23:21:49.166787 5491 leader_election.cc:302] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27efe785ba7745638ef2aa16a2fd0e14; no voters:
I20260926 23:21:49.167418 5491 leader_election.cc:288] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:49.167584 5501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:49.168372 5501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [term 1 LEADER]: Becoming Leader. State: Replica: 27efe785ba7745638ef2aa16a2fd0e14, State: Running, Role: LEADER
I20260926 23:21:49.168890 5501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27efe785ba7745638ef2aa16a2fd0e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260926 23:21:49.169526 5491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:49.171834 5502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27efe785ba7745638ef2aa16a2fd0e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27efe785ba7745638ef2aa16a2fd0e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260926 23:21:49.172338 5502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:49.172302 5503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27efe785ba7745638ef2aa16a2fd0e14. Latest consensus state: current_term: 1 leader_uuid: "27efe785ba7745638ef2aa16a2fd0e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27efe785ba7745638ef2aa16a2fd0e14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260926 23:21:49.172605 5503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:49.173900 5511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:49.178025 5511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:49.186511 5511 catalog_manager.cc:1347] Generated new cluster ID: 8a2e62619b414879a2074cf8509733c9
I20260926 23:21:49.186616 5511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:49.209952 5511 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:49.210374 5511 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:49.231410 5511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27efe785ba7745638ef2aa16a2fd0e14: Generated new TSK 0
I20260926 23:21:49.232357 5511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:49.240712 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5424
W20260926 23:21:49.242094 5486 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.106:57119
W20260926 23:21:49.242758 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5424: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:21:49.242901 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 5424
2026-09-26T23:21:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:49 dist-test-slave-vcv1 krb5kdc[5522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:49 dist-test-slave-vcv1 krb5kdc[5522](info): set up 2 sockets
Sep 26 23:21:49 dist-test-slave-vcv1 krb5kdc[5522](info): commencing operation
krb5kdc: starting...
W20260926 23:21:50.345577 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:50 dist-test-slave-vcv1 krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:50.591059 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:50.912834 5535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:50.913110 5535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:50.913203 5535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:50.920203 5535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:50.920303 5535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:50.920356 5535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:50.920416 5535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:50.920460 5535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:50.930130 5535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:50.931299 5535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:50.932519 5535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:50.943903 5541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:50.943986 5540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:50.943933 5543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:50.949576 5535 server_base.cc:1034] running on GCE node
I20260926 23:21:50.950332 5535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:50.951589 5535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:50.952790 5535 hybrid_clock.cc:648] HybridClock initialized: now 1790464910952731 us; error 47 us; skew 500 ppm
I20260926 23:21:50.953085 5535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:50 dist-test-slave-vcv1 krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:50.956009 5535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:50.957402 5535 webserver.cc:460] Webserver started at http://127.0.0.1:43167/ using document root <none> and password file <none>
I20260926 23:21:50.958030 5535 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:50.958134 5535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:50.958401 5535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:50.961339 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "ffa748727bda42a1841bf9131f979fdf"
format_stamp: "Formatted at 2026-09-26 23:21:50 on dist-test-slave-vcv1"
server_key: "0dec661e8e3e8edd3de8ca96c190ee3b"
server_key_iv: "a66dfda267e5dba327b17c7ce2d9147f"
server_key_version: "encryptionkey@0"
I20260926 23:21:50.962020 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "ffa748727bda42a1841bf9131f979fdf"
format_stamp: "Formatted at 2026-09-26 23:21:50 on dist-test-slave-vcv1"
server_key: "0dec661e8e3e8edd3de8ca96c190ee3b"
server_key_iv: "a66dfda267e5dba327b17c7ce2d9147f"
server_key_version: "encryptionkey@0"
I20260926 23:21:50.966852 5535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:21:50.970172 5550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:50.971431 5535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:21:50.971563 5535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "ffa748727bda42a1841bf9131f979fdf"
format_stamp: "Formatted at 2026-09-26 23:21:50 on dist-test-slave-vcv1"
server_key: "0dec661e8e3e8edd3de8ca96c190ee3b"
server_key_iv: "a66dfda267e5dba327b17c7ce2d9147f"
server_key_version: "encryptionkey@0"
I20260926 23:21:50.971715 5535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:50.993373 5535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:50.997752 5535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:50.997990 5535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:51.016119 5535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41379
I20260926 23:21:51.016197 5601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41379 every 8 connection(s)
I20260926 23:21:51.017491 5535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:51.021003 5602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:51.023716 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 5535
I20260926 23:21:51.023900 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:51.024415 16633 external_mini_cluster.cc:1428] Setting key 27c64c34a414a4f717c2e0bcebbac411
I20260926 23:21:51.032951 5602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf: Bootstrap starting.
I20260926 23:21:51.036685 5602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:51.037987 5602 log.cc:824] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:21:51 dist-test-slave-vcv1 krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:21:51.040891 5602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf: No bootstrap required, opened a new log
I20260926 23:21:51.047874 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:51.026202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:47757 (local address 127.0.0.1:41379)
0926 23:21:51.027014 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:51.027025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:51.027065 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:21:51.028821 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:51.028844 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:51.029251 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:51.029584 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:51.031436 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:51.032769 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:51.034517 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:51.035167 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:51.035302 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:51.040291 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:51.040298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:51.040311 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:51.040345 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:51.043603 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:51.044413 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:51.044419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:51.044425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:51.044584 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:51.045107 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:51.045111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:51.045113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:51.045603 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:51.045953 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:51.046445 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 23:21:51.046870 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":385,"threads_started":1}
I20260926 23:21:51.048887 5602 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa748727bda42a1841bf9131f979fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } }
I20260926 23:21:51.049320 5602 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:51.049430 5602 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa748727bda42a1841bf9131f979fdf, State: Initialized, Role: FOLLOWER
I20260926 23:21:51.050135 5602 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa748727bda42a1841bf9131f979fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } }
I20260926 23:21:51.050514 5602 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:51.050670 5602 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:51.050843 5602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:51.052228 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:51.053406 5602 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa748727bda42a1841bf9131f979fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } }
I20260926 23:21:51.053885 5602 leader_election.cc:302] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa748727bda42a1841bf9131f979fdf; no voters:
I20260926 23:21:51.054543 5602 leader_election.cc:288] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:51.054638 5607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:51.061448 5607 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [term 1 LEADER]: Becoming Leader. State: Replica: ffa748727bda42a1841bf9131f979fdf, State: Running, Role: LEADER
I20260926 23:21:51.062073 5607 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa748727bda42a1841bf9131f979fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } }
I20260926 23:21:51.063032 5602 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:51.064972 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa748727bda42a1841bf9131f979fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa748727bda42a1841bf9131f979fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } } }
I20260926 23:21:51.065430 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:51.065955 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa748727bda42a1841bf9131f979fdf. Latest consensus state: current_term: 1 leader_uuid: "ffa748727bda42a1841bf9131f979fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa748727bda42a1841bf9131f979fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } } }
I20260926 23:21:51.066691 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:51.078845 5617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:51.080986 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:51.056750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:32771 (local address 127.0.0.1:41379)
0926 23:21:51.056904 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:51.056908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:51.060728 (+ 3820us) server_negotiation.cc:407] Connection header received
0926 23:21:51.060930 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:51.060935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:51.061011 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:51.061121 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:51.062820 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:51.063952 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:51.065917 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:51.066591 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:51.066736 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:51.071025 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:51.071037 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:51.071044 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:51.071099 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:51.076007 (+ 4908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:51.077338 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:51.077346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:51.077356 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:51.077540 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:51.078224 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:51.078229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:51.078232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:51.078667 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:51.078945 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:51.080196 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0926 23:21:51.080438 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":38}
I20260926 23:21:51.084075 5617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:51.094026 5617 catalog_manager.cc:1347] Generated new cluster ID: 3bdc3fead4cb464390aacaf14d4180cd
I20260926 23:21:51.094141 5617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:51.109993 5617 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:51.111188 5617 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:51.119252 5617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa748727bda42a1841bf9131f979fdf: Generated new TSK 0
I20260926 23:21:51.120008 5617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:51.129904 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5535
I20260926 23:21:51.131214 5597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:51.221064 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 5535
W20260926 23:21:51.239924 5520 connection.cc:535] client connection to 127.0.0.1:41379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41379): Connection reset by peer (error 104)
2026-09-26T23:21:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1979 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:51 dist-test-slave-vcv1 krb5kdc[5634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:51 dist-test-slave-vcv1 krb5kdc[5634](info): set up 2 sockets
Sep 26 23:21:51 dist-test-slave-vcv1 krb5kdc[5634](info): commencing operation
krb5kdc: starting...
W20260926 23:21:52.325961 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:52 dist-test-slave-vcv1 krb5kdc[5634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:52.569612 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:52.905889 5647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:52.906226 5647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:52.906394 5647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:52.913586 5647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:52.913733 5647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:52.913833 5647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:52.913939 5647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:52.914021 5647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:52.923627 5647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:52.924919 5647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:52.926186 5647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:52.937366 5655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:52.937463 5653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:52.937413 5652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:52.938570 5647 server_base.cc:1034] running on GCE node
I20260926 23:21:52.939285 5647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:52.940634 5647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:52.941869 5647 hybrid_clock.cc:648] HybridClock initialized: now 1790464912941832 us; error 34 us; skew 500 ppm
I20260926 23:21:52.942205 5647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:52 dist-test-slave-vcv1 krb5kdc[5634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:52.945160 5647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:52.946504 5647 webserver.cc:460] Webserver started at http://127.0.0.1:41929/ using document root <none> and password file <none>
I20260926 23:21:52.947151 5647 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:52.947305 5647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:52.947615 5647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:52.950520 5647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "f1b0f5cb03af4cf58b4bb00758fb1414"
format_stamp: "Formatted at 2026-09-26 23:21:52 on dist-test-slave-vcv1"
server_key: "40d0f686c1235fc22923c03a8db89527"
server_key_iv: "d3e5edaf44637c27d5fb4ccb965965cd"
server_key_version: "encryptionkey@0"
I20260926 23:21:52.951269 5647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "f1b0f5cb03af4cf58b4bb00758fb1414"
format_stamp: "Formatted at 2026-09-26 23:21:52 on dist-test-slave-vcv1"
server_key: "40d0f686c1235fc22923c03a8db89527"
server_key_iv: "d3e5edaf44637c27d5fb4ccb965965cd"
server_key_version: "encryptionkey@0"
I20260926 23:21:52.956046 5647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 23:21:52.959214 5662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:52.960460 5647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 23:21:52.960644 5647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "f1b0f5cb03af4cf58b4bb00758fb1414"
format_stamp: "Formatted at 2026-09-26 23:21:52 on dist-test-slave-vcv1"
server_key: "40d0f686c1235fc22923c03a8db89527"
server_key_iv: "d3e5edaf44637c27d5fb4ccb965965cd"
server_key_version: "encryptionkey@0"
I20260926 23:21:52.960829 5647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:52.978680 5647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:52.982719 5647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:52.982998 5647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:53.001286 5647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43859
I20260926 23:21:53.001325 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43859 every 8 connection(s)
I20260926 23:21:53.002622 5647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:53.006052 5714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:53.009182 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 5647
I20260926 23:21:53.009359 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:53.009788 16633 external_mini_cluster.cc:1428] Setting key 6afadcaceb0975e80309ea10a792bf0d
I20260926 23:21:53.018843 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414: Bootstrap starting.
I20260926 23:21:53.022675 5714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:53.023977 5714 log.cc:824] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:53.024682 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:53.011470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:34465 (local address 127.0.0.1:43859)
0926 23:21:53.012219 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:53.012233 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:53.012280 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:21:53.014474 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:53.014502 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:53.015009 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:53.015432 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:53.015940 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:53.017994 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:53.019434 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:53.020499 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:53.020671 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:53.021866 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:53.021876 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:53.021893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:53.022565 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:53.022664 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:53.022703 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:53.022856 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:53.023203 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 23:21:53.023356 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":309,"threads_started":1}
I20260926 23:21:53.026767 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414: No bootstrap required, opened a new log
I20260926 23:21:53.028337 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:53.036825 5714 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0f5cb03af4cf58b4bb00758fb1414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } }
I20260926 23:21:53.037165 5714 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:53.037294 5714 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b0f5cb03af4cf58b4bb00758fb1414, State: Initialized, Role: FOLLOWER
I20260926 23:21:53.037950 5714 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0f5cb03af4cf58b4bb00758fb1414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } }
I20260926 23:21:53.038234 5714 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:53.038388 5714 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:53.038578 5714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:53.040318 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:53.033479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:34743 (local address 127.0.0.1:43859)
0926 23:21:53.033652 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:53.033658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:53.034097 (+ 439us) server_negotiation.cc:407] Connection header received
0926 23:21:53.034256 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:53.034261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:53.034345 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:53.034450 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:53.034932 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:53.036101 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:53.037574 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:53.038254 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:53.038384 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:53.039564 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:53.039571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:53.039575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:53.039595 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:53.039642 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:53.039662 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:53.039779 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:53.039901 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 23:21:53.039955 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:21:53.041716 5714 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0f5cb03af4cf58b4bb00758fb1414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } }
I20260926 23:21:53.042129 5714 leader_election.cc:302] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1b0f5cb03af4cf58b4bb00758fb1414; no voters:
I20260926 23:21:53.042701 5714 leader_election.cc:288] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:53.042860 5724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:53.043591 5724 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [term 1 LEADER]: Becoming Leader. State: Replica: f1b0f5cb03af4cf58b4bb00758fb1414, State: Running, Role: LEADER
I20260926 23:21:53.044252 5724 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0f5cb03af4cf58b4bb00758fb1414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } }
I20260926 23:21:53.044771 5714 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:53.046669 5726 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1b0f5cb03af4cf58b4bb00758fb1414. Latest consensus state: current_term: 1 leader_uuid: "f1b0f5cb03af4cf58b4bb00758fb1414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0f5cb03af4cf58b4bb00758fb1414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } } }
I20260926 23:21:53.046977 5726 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:53.047804 5729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:53.047695 5725 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1b0f5cb03af4cf58b4bb00758fb1414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0f5cb03af4cf58b4bb00758fb1414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } } }
I20260926 23:21:53.048705 5725 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:53.053355 5729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:53.065383 5729 catalog_manager.cc:1347] Generated new cluster ID: 63fb6e3c9f874835902ee64b0e45431d
I20260926 23:21:53.065505 5729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:53.087775 5729 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:53.089112 5729 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:53.094893 5729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1b0f5cb03af4cf58b4bb00758fb1414: Generated new TSK 0
I20260926 23:21:53.095739 5729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:53.101177 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5647
W20260926 23:21:53.103107 5709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.106:34465
W20260926 23:21:53.103856 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5647: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:21:53.103992 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 5647
W20260926 23:21:53.120584 5632 connection.cc:535] client connection to 127.0.0.1:43859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43859): Connection reset by peer (error 104)
2026-09-26T23:21:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:53 dist-test-slave-vcv1 krb5kdc[5745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:53 dist-test-slave-vcv1 krb5kdc[5745](info): set up 2 sockets
Sep 26 23:21:53 dist-test-slave-vcv1 krb5kdc[5745](info): commencing operation
krb5kdc: starting...
W20260926 23:21:54.205466 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:54 dist-test-slave-vcv1 krb5kdc[5745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:21:54.447510 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:54.770076 5758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:54.770339 5758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:54.770455 5758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:54.777524 5758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:54.777626 5758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:54.777670 5758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:54.777748 5758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:54.777823 5758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:54.788241 5758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:54.789479 5758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:54.790730 5758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:54.802829 5766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:54.802876 5764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:54.802898 5763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:54.803727 5758 server_base.cc:1034] running on GCE node
I20260926 23:21:54.804556 5758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:54.805898 5758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:54.807104 5758 hybrid_clock.cc:648] HybridClock initialized: now 1790464914807055 us; error 39 us; skew 500 ppm
I20260926 23:21:54.807442 5758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:54 dist-test-slave-vcv1 krb5kdc[5745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:54.814642 5758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:21:54.816319 5758 webserver.cc:460] Webserver started at http://127.0.0.1:45593/ using document root <none> and password file <none>
I20260926 23:21:54.816972 5758 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:54.817123 5758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:54.817442 5758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:54.820380 5758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "8cd088c50eec44d9b048fd598948111a"
format_stamp: "Formatted at 2026-09-26 23:21:54 on dist-test-slave-vcv1"
server_key: "a1ff68158694d9df9a8e67cd2e71da4c"
server_key_iv: "d35912c9f38d9025c4d8d4c24d4fd19d"
server_key_version: "encryptionkey@0"
I20260926 23:21:54.821138 5758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "8cd088c50eec44d9b048fd598948111a"
format_stamp: "Formatted at 2026-09-26 23:21:54 on dist-test-slave-vcv1"
server_key: "a1ff68158694d9df9a8e67cd2e71da4c"
server_key_iv: "d35912c9f38d9025c4d8d4c24d4fd19d"
server_key_version: "encryptionkey@0"
I20260926 23:21:54.826272 5758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 23:21:54.829546 5773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:54.830960 5758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:21:54.831161 5758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "8cd088c50eec44d9b048fd598948111a"
format_stamp: "Formatted at 2026-09-26 23:21:54 on dist-test-slave-vcv1"
server_key: "a1ff68158694d9df9a8e67cd2e71da4c"
server_key_iv: "d35912c9f38d9025c4d8d4c24d4fd19d"
server_key_version: "encryptionkey@0"
I20260926 23:21:54.831357 5758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:54.836999 5758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:54.840914 5758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:54.841208 5758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:54.859993 5758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44203
I20260926 23:21:54.859992 5824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44203 every 8 connection(s)
I20260926 23:21:54.861424 5758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:54.864913 5825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:54.870205 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 5758
I20260926 23:21:54.870388 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:54.870839 16633 external_mini_cluster.cc:1428] Setting key 8bd5423facbef3f5b0a44de7045bf066
I20260926 23:21:54.875232 5825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a: Bootstrap starting.
I20260926 23:21:54.878257 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:54.872527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:51829 (local address 127.0.0.1:44203)
0926 23:21:54.873329 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:54.873339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:54.873380 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:21:54.874966 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:54.874987 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:54.875292 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:54.875711 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:54.876774 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:54.876780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:54.876792 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:54.876811 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:54.876861 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:54.876882 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:54.877120 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:54.877251 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 23:21:54.877417 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":351,"threads_started":1}
I20260926 23:21:54.878472 5825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:54.879712 5825 log.cc:824] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:54.882017 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:54.882743 5825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a: No bootstrap required, opened a new log
I20260926 23:21:54.889341 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:54.887101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:59907 (local address 127.0.0.1:44203)
0926 23:21:54.887260 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:54.887265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:54.887531 (+ 266us) server_negotiation.cc:407] Connection header received
0926 23:21:54.887658 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:54.887668 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:54.887728 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:54.887833 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:54.888710 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:54.888716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:54.888720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:54.888739 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:54.888786 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:54.888805 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:54.888931 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:54.888989 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 23:21:54.889049 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:21:54.891986 5825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd088c50eec44d9b048fd598948111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } }
I20260926 23:21:54.892416 5825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:54.892522 5825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd088c50eec44d9b048fd598948111a, State: Initialized, Role: FOLLOWER
I20260926 23:21:54.893047 5825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd088c50eec44d9b048fd598948111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } }
I20260926 23:21:54.893282 5825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:54.893431 5825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:54.893608 5825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:54.895720 5825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd088c50eec44d9b048fd598948111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } }
I20260926 23:21:54.896257 5825 leader_election.cc:302] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd088c50eec44d9b048fd598948111a; no voters:
I20260926 23:21:54.896826 5825 leader_election.cc:288] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:54.897011 5835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:54.898383 5835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [term 1 LEADER]: Becoming Leader. State: Replica: 8cd088c50eec44d9b048fd598948111a, State: Running, Role: LEADER
I20260926 23:21:54.898909 5825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:54.898873 5835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd088c50eec44d9b048fd598948111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } }
I20260926 23:21:54.901964 5837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cd088c50eec44d9b048fd598948111a. Latest consensus state: current_term: 1 leader_uuid: "8cd088c50eec44d9b048fd598948111a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd088c50eec44d9b048fd598948111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } } }
I20260926 23:21:54.902447 5836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cd088c50eec44d9b048fd598948111a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd088c50eec44d9b048fd598948111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } } }
I20260926 23:21:54.902724 5837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:54.902832 5836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:54.912523 5844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:54.916468 5844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:54.923759 5844 catalog_manager.cc:1347] Generated new cluster ID: 96a313c0ba9f46d6bf2644a1a9f79047
I20260926 23:21:54.923858 5844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:54.940227 5844 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:54.940620 5844 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:54.949472 5844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cd088c50eec44d9b048fd598948111a: Generated new TSK 0
I20260926 23:21:54.950183 5844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:54.961591 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5758
W20260926 23:21:54.962739 5820 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.106:51829
W20260926 23:21:54.963372 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5758: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:21:54.963524 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 5758
2026-09-26T23:21:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:55 dist-test-slave-vcv1 krb5kdc[5856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:55 dist-test-slave-vcv1 krb5kdc[5856](info): set up 2 sockets
Sep 26 23:21:55 dist-test-slave-vcv1 krb5kdc[5856](info): commencing operation
krb5kdc: starting...
W20260926 23:21:56.057579 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:56 dist-test-slave-vcv1 krb5kdc[5856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:21:56.305912 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:34467
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:34467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:56.633595 5869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:56.633850 5869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:56.633948 5869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:56.641068 5869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:56.641177 5869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:56.641222 5869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:56.641283 5869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:56.641326 5869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:56.650964 5869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:34467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:34467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:56.652156 5869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:56.653416 5869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:56.665158 5877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:56.665202 5875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:56.665474 5874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:56.666306 5869 server_base.cc:1034] running on GCE node
I20260926 23:21:56.667114 5869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:56.668601 5869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:56.669818 5869 hybrid_clock.cc:648] HybridClock initialized: now 1790464916669761 us; error 51 us; skew 500 ppm
I20260926 23:21:56.670193 5869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:56 dist-test-slave-vcv1 krb5kdc[5856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:56.673362 5869 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:21:56.674849 5869 webserver.cc:460] Webserver started at http://127.16.62.126:46819/ using document root <none> and password file <none>
I20260926 23:21:56.675516 5869 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:56.675678 5869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:56.675987 5869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:56.678915 5869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "5d61250565c7402d83c0d2955ae63aea"
format_stamp: "Formatted at 2026-09-26 23:21:56 on dist-test-slave-vcv1"
server_key: "c0cc97796019ea24e2fad1781f53d3ec"
server_key_iv: "7adc2705e2a56cd09d672bb276e1ac02"
server_key_version: "encryptionkey@0"
I20260926 23:21:56.679812 5869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "5d61250565c7402d83c0d2955ae63aea"
format_stamp: "Formatted at 2026-09-26 23:21:56 on dist-test-slave-vcv1"
server_key: "c0cc97796019ea24e2fad1781f53d3ec"
server_key_iv: "7adc2705e2a56cd09d672bb276e1ac02"
server_key_version: "encryptionkey@0"
I20260926 23:21:56.685164 5869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:21:56.688570 5884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:56.689815 5869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 23:21:56.690019 5869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "5d61250565c7402d83c0d2955ae63aea"
format_stamp: "Formatted at 2026-09-26 23:21:56 on dist-test-slave-vcv1"
server_key: "c0cc97796019ea24e2fad1781f53d3ec"
server_key_iv: "7adc2705e2a56cd09d672bb276e1ac02"
server_key_version: "encryptionkey@0"
I20260926 23:21:56.690207 5869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:56.709858 5869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:56.714133 5869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:56.714406 5869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:56.732486 5869 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:34467
I20260926 23:21:56.732504 5936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:34467 every 8 connection(s)
I20260926 23:21:56.733809 5869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:56.737315 5937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:56.740831 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 5869
I20260926 23:21:56.741000 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:56.741402 16633 external_mini_cluster.cc:1428] Setting key eae6bd534a33c00ec8d0fb523579f9c6
I20260926 23:21:56.748636 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea: Bootstrap starting.
I20260926 23:21:56.751513 5937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:56.752537 5937 log.cc:824] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:21:56 dist-test-slave-vcv1 krb5kdc[5856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:21:56.755673 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea: No bootstrap required, opened a new log
I20260926 23:21:56.762982 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:56.743014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33817 (local address 127.16.62.126:34467)
0926 23:21:56.743696 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:56.743707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:56.743745 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:21:56.745392 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:56.745414 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:56.745787 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:56.746176 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:56.747557 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:56.748952 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:56.750155 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:56.750780 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:56.750963 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:56.755829 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:56.755837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:56.755851 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:56.755886 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:56.759030 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:56.759901 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:56.759907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:56.759916 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:56.760067 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:56.760549 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:56.760553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:56.760555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:56.761036 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:56.761266 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:56.761719 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 23:21:56.762015 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":300,"threads_started":1}
I20260926 23:21:56.764295 5937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d61250565c7402d83c0d2955ae63aea" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34467 } }
I20260926 23:21:56.764658 5937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:56.764765 5937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d61250565c7402d83c0d2955ae63aea, State: Initialized, Role: FOLLOWER
I20260926 23:21:56.765359 5937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d61250565c7402d83c0d2955ae63aea" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34467 } }
I20260926 23:21:56.765652 5937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:56.765841 5937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:56.766036 5937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:56.766793 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:56.768379 5937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d61250565c7402d83c0d2955ae63aea" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34467 } }
I20260926 23:21:56.768826 5937 leader_election.cc:302] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d61250565c7402d83c0d2955ae63aea; no voters:
I20260926 23:21:56.769426 5937 leader_election.cc:288] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:56.769516 5943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:56.771029 5943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [term 1 LEADER]: Becoming Leader. State: Replica: 5d61250565c7402d83c0d2955ae63aea, State: Running, Role: LEADER
I20260926 23:21:56.771657 5937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:56.771682 5943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d61250565c7402d83c0d2955ae63aea" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34467 } }
I20260926 23:21:56.777426 5945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d61250565c7402d83c0d2955ae63aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d61250565c7402d83c0d2955ae63aea" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34467 } } }
I20260926 23:21:56.777786 5945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:56.778260 5947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d61250565c7402d83c0d2955ae63aea. Latest consensus state: current_term: 1 leader_uuid: "5d61250565c7402d83c0d2955ae63aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d61250565c7402d83c0d2955ae63aea" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34467 } } }
I20260926 23:21:56.778551 5947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:56.798056 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:56.772672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44503 (local address 127.16.62.126:34467)
0926 23:21:56.772819 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:56.772825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:56.778900 (+ 6075us) server_negotiation.cc:407] Connection header received
0926 23:21:56.779127 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:56.779135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:56.779234 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:56.779347 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:56.781904 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:56.783382 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:56.785872 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:56.786824 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:56.786941 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:56.791061 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:56.791071 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:56.791079 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:21:56.791138 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:56.795021 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:56.795828 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:56.795834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:56.795836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:56.795942 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:21:56.796466 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:21:56.796469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:21:56.796471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:21:56.796790 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:56.797073 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:56.797462 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 23:21:56.797633 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 23:21:56.801442 5959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:56.806020 5959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:56.814471 5959 catalog_manager.cc:1347] Generated new cluster ID: a9cee8db17424fd1b241e8b758be78b6
I20260926 23:21:56.814637 5959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:56.840884 5959 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:56.842137 5959 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:56.849356 5959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d61250565c7402d83c0d2955ae63aea: Generated new TSK 0
I20260926 23:21:56.850133 5959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:56.861972 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5869
I20260926 23:21:56.863127 5932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:21:56.956943 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 5869
W20260926 23:21:56.974583 5854 connection.cc:535] client connection to 127.16.62.126:34467 recv error: Network error: failed to read from TLS socket (remote: 127.16.62.126:34467): Connection reset by peer (error 104)
2026-09-26T23:21:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:57 dist-test-slave-vcv1 krb5kdc[5970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:57 dist-test-slave-vcv1 krb5kdc[5970](info): set up 2 sockets
Sep 26 23:21:57 dist-test-slave-vcv1 krb5kdc[5970](info): commencing operation
krb5kdc: starting...
W20260926 23:21:58.053511 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:21:58 dist-test-slave-vcv1 krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:21:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:21:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:21:58.294900 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:40745
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:21:58.613229 5983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:21:58.613556 5983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:21:58.613727 5983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:21:58.620936 5983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:21:58.621109 5983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:21:58.621213 5983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:21:58.621321 5983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:21:58.621397 5983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:21:58.631160 5983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:40745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:21:58.632503 5983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:21:58.633790 5983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:21:58.645471 5989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:58.645428 5991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:21:58.645511 5988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:21:58.646246 5983 server_base.cc:1034] running on GCE node
I20260926 23:21:58.647033 5983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:21:58.648486 5983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:21:58.649708 5983 hybrid_clock.cc:648] HybridClock initialized: now 1790464918649667 us; error 34 us; skew 500 ppm
I20260926 23:21:58.650038 5983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:21:58 dist-test-slave-vcv1 krb5kdc[5970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464918, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:21:58.653025 5983 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:21:58.654404 5983 webserver.cc:460] Webserver started at http://127.16.62.126:45911/ using document root <none> and password file <none>
I20260926 23:21:58.655069 5983 fs_manager.cc:359] Metadata directory not provided
I20260926 23:21:58.655212 5983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:21:58.655509 5983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:21:58.658452 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "490361a076d44dbc8d95a202688b4e01"
format_stamp: "Formatted at 2026-09-26 23:21:58 on dist-test-slave-vcv1"
server_key: "108662fbfa4af1240ac25009de657a62"
server_key_iv: "28d39e1c98e80012dfc699f079100a2b"
server_key_version: "encryptionkey@0"
I20260926 23:21:58.659214 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "490361a076d44dbc8d95a202688b4e01"
format_stamp: "Formatted at 2026-09-26 23:21:58 on dist-test-slave-vcv1"
server_key: "108662fbfa4af1240ac25009de657a62"
server_key_iv: "28d39e1c98e80012dfc699f079100a2b"
server_key_version: "encryptionkey@0"
I20260926 23:21:58.664096 5983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 23:21:58.667598 5998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:21:58.668867 5983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:21:58.669050 5983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "490361a076d44dbc8d95a202688b4e01"
format_stamp: "Formatted at 2026-09-26 23:21:58 on dist-test-slave-vcv1"
server_key: "108662fbfa4af1240ac25009de657a62"
server_key_iv: "28d39e1c98e80012dfc699f079100a2b"
server_key_version: "encryptionkey@0"
I20260926 23:21:58.669241 5983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:21:58.683276 5983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:21:58.687242 5983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:21:58.687637 5983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:21:58.705415 5983 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:40745
I20260926 23:21:58.705435 6050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:40745 every 8 connection(s)
I20260926 23:21:58.706754 5983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:21:58.707692 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 5983
I20260926 23:21:58.707863 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:21:58.708390 16633 external_mini_cluster.cc:1428] Setting key 3aac48d1d060db0e20e87a23f44f5048
I20260926 23:21:58.711035 6051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:21:58.722500 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:58.710069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44261 (local address 127.16.62.126:40745)
0926 23:21:58.710819 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:58.710836 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:58.710882 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:21:58.713200 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:58.713224 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:58.713695 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:58.714108 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:58.714813 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:58.716384 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:58.718184 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:58.718814 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:58.718940 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:58.720253 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:58.720261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:58.720274 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:58.720824 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:58.720900 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:58.720931 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:58.721051 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:58.721365 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 23:21:58.721559 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":310,"threads_started":1}
I20260926 23:21:58.723732 6051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01: Bootstrap starting.
I20260926 23:21:58.726508 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:21:58.727427 6051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01: Neither blocks nor log segments found. Creating new log.
I20260926 23:21:58.728644 6051 log.cc:824] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01: Log is configured to *not* fsync() on all Append() calls
I20260926 23:21:58.731884 6051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01: No bootstrap required, opened a new log
I20260926 23:21:58.739703 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:21:58.731648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40097 (local address 127.16.62.126:40745)
0926 23:21:58.731818 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 23:21:58.731824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:21:58.732352 (+ 528us) server_negotiation.cc:407] Connection header received
0926 23:21:58.732490 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:21:58.732495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:21:58.732589 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:21:58.732696 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:21:58.733209 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:58.734379 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:58.736004 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:21:58.736852 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:21:58.737020 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:21:58.738715 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:21:58.738722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:21:58.738727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:21:58.738747 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:21:58.738806 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:21:58.738831 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:21:58.738952 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:21:58.739207 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 23:21:58.739265 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:21:58.743712 6051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490361a076d44dbc8d95a202688b4e01" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40745 } }
I20260926 23:21:58.744103 6051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:21:58.744235 6051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490361a076d44dbc8d95a202688b4e01, State: Initialized, Role: FOLLOWER
I20260926 23:21:58.744881 6051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490361a076d44dbc8d95a202688b4e01" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40745 } }
I20260926 23:21:58.745179 6051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:21:58.745332 6051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:21:58.745513 6051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:21:58.747649 6051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490361a076d44dbc8d95a202688b4e01" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40745 } }
I20260926 23:21:58.748184 6051 leader_election.cc:302] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 490361a076d44dbc8d95a202688b4e01; no voters:
I20260926 23:21:58.748756 6051 leader_election.cc:288] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:21:58.748886 6061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:21:58.750192 6061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [term 1 LEADER]: Becoming Leader. State: Replica: 490361a076d44dbc8d95a202688b4e01, State: Running, Role: LEADER
I20260926 23:21:58.750732 6061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490361a076d44dbc8d95a202688b4e01" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40745 } }
I20260926 23:21:58.750986 6051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:21:58.753679 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 490361a076d44dbc8d95a202688b4e01. Latest consensus state: current_term: 1 leader_uuid: "490361a076d44dbc8d95a202688b4e01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490361a076d44dbc8d95a202688b4e01" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40745 } } }
I20260926 23:21:58.753608 6062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "490361a076d44dbc8d95a202688b4e01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490361a076d44dbc8d95a202688b4e01" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40745 } } }
I20260926 23:21:58.754029 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:58.754076 6062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01 [sys.catalog]: This master's current role is: LEADER
I20260926 23:21:58.755556 6070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:21:58.760855 6070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:21:58.769757 6070 catalog_manager.cc:1347] Generated new cluster ID: 88e5b0fd29e54ed0b4453793cf5e2a05
I20260926 23:21:58.769872 6070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:21:58.793149 6070 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:21:58.794533 6070 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:21:58.807448 6070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 490361a076d44dbc8d95a202688b4e01: Generated new TSK 0
I20260926 23:21:58.808271 6070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:21:58.822268 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5983
W20260926 23:21:58.823640 6046 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44261
W20260926 23:21:58.824398 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 5983: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:21:58.824548 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 5983
W20260926 23:21:58.841454 5968 connection.cc:535] client connection to 127.16.62.126:40745 recv error: Network error: failed to read from TLS socket (remote: 127.16.62.126:40745): Connection reset by peer (error 104)
2026-09-26T23:21:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:21:58 dist-test-slave-vcv1 krb5kdc[6082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:21:58 dist-test-slave-vcv1 krb5kdc[6082](info): set up 2 sockets
Sep 26 23:21:58 dist-test-slave-vcv1 krb5kdc[6082](info): commencing operation
krb5kdc: starting...
W20260926 23:21:59.921578 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:22:00 dist-test-slave-vcv1 krb5kdc[6082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:22:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:22:00.171386 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:33681
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:22:00.488231 6095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:00.488543 6095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:00.488708 6095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:00.495909 6095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:00.496057 6095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:00.496183 6095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:00.496284 6095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:00.496369 6095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:00.505712 6095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:33681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:00.506937 6095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:00.508234 6095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:22:00.520355 6100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:00.520280 6103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:00.520315 6101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:00.521117 6095 server_base.cc:1034] running on GCE node
I20260926 23:22:00.521772 6095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:00.523041 6095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:00.524240 6095 hybrid_clock.cc:648] HybridClock initialized: now 1790464920524191 us; error 48 us; skew 500 ppm
I20260926 23:22:00.524578 6095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:00 dist-test-slave-vcv1 krb5kdc[6082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464920, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:00.532658 6095 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:22:00.534168 6095 webserver.cc:460] Webserver started at http://127.16.62.126:37541/ using document root <none> and password file <none>
I20260926 23:22:00.534830 6095 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:00.534972 6095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:00.535279 6095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:00.538314 6095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "17c66ba8730f41eea49e05e5247d77b0"
format_stamp: "Formatted at 2026-09-26 23:22:00 on dist-test-slave-vcv1"
server_key: "f746f73edf7710cd1690079112016236"
server_key_iv: "a8c49b41921fa32fc866d71d78c5c3e4"
server_key_version: "encryptionkey@0"
I20260926 23:22:00.539048 6095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "17c66ba8730f41eea49e05e5247d77b0"
format_stamp: "Formatted at 2026-09-26 23:22:00 on dist-test-slave-vcv1"
server_key: "f746f73edf7710cd1690079112016236"
server_key_iv: "a8c49b41921fa32fc866d71d78c5c3e4"
server_key_version: "encryptionkey@0"
I20260926 23:22:00.543958 6095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 23:22:00.547225 6110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:00.548600 6095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:22:00.548768 6095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "17c66ba8730f41eea49e05e5247d77b0"
format_stamp: "Formatted at 2026-09-26 23:22:00 on dist-test-slave-vcv1"
server_key: "f746f73edf7710cd1690079112016236"
server_key_iv: "a8c49b41921fa32fc866d71d78c5c3e4"
server_key_version: "encryptionkey@0"
I20260926 23:22:00.548965 6095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:00.554437 6095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:00.558223 6095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:00.558506 6095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:00.576646 6095 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:33681
I20260926 23:22:00.576691 6162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:33681 every 8 connection(s)
I20260926 23:22:00.577971 6095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:22:00.581437 6163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:00.582695 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 6095
I20260926 23:22:00.582885 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:22:00.583436 16633 external_mini_cluster.cc:1428] Setting key dd6cdd14f55d3ae73cba2dbb382b481c
I20260926 23:22:00.590847 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:00.585197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60919 (local address 127.16.62.126:33681)
0926 23:22:00.586004 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:00.586015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:00.586058 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:22:00.587673 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:00.587695 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:00.587998 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:00.588368 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:00.589387 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:00.589393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:00.589405 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:22:00.589419 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:00.589476 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:22:00.589494 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:00.589659 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:00.589807 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 23:22:00.590003 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":388,"threads_started":1}
I20260926 23:22:00.594406 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:22:00.594874 6163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0: Bootstrap starting.
I20260926 23:22:00.597867 6163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:00.599053 6163 log.cc:824] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0: Log is configured to *not* fsync() on all Append() calls
I20260926 23:22:00.601164 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:00.598677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35701 (local address 127.16.62.126:33681)
0926 23:22:00.598897 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:00.598903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:00.599379 (+ 476us) server_negotiation.cc:407] Connection header received
0926 23:22:00.599503 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:00.599511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:00.599598 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:00.599719 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:00.600578 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:00.600593 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:00.600597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:22:00.600613 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:00.600657 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:22:00.600676 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:00.600788 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:00.600856 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 23:22:00.600906 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:22:00.602295 6163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0: No bootstrap required, opened a new log
I20260926 23:22:00.609407 6163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c66ba8730f41eea49e05e5247d77b0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33681 } }
I20260926 23:22:00.609764 6163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:22:00.609869 6163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c66ba8730f41eea49e05e5247d77b0, State: Initialized, Role: FOLLOWER
I20260926 23:22:00.610457 6163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c66ba8730f41eea49e05e5247d77b0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33681 } }
I20260926 23:22:00.610747 6163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:22:00.610883 6163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:22:00.611040 6163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:00.613171 6163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c66ba8730f41eea49e05e5247d77b0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33681 } }
I20260926 23:22:00.613642 6163 leader_election.cc:302] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c66ba8730f41eea49e05e5247d77b0; no voters:
I20260926 23:22:00.614213 6163 leader_election.cc:288] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:22:00.614336 6173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:22:00.614972 6173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [term 1 LEADER]: Becoming Leader. State: Replica: 17c66ba8730f41eea49e05e5247d77b0, State: Running, Role: LEADER
I20260926 23:22:00.615528 6173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c66ba8730f41eea49e05e5247d77b0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33681 } }
I20260926 23:22:00.616274 6163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:22:00.618501 6174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17c66ba8730f41eea49e05e5247d77b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c66ba8730f41eea49e05e5247d77b0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33681 } } }
I20260926 23:22:00.618462 6175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17c66ba8730f41eea49e05e5247d77b0. Latest consensus state: current_term: 1 leader_uuid: "17c66ba8730f41eea49e05e5247d77b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c66ba8730f41eea49e05e5247d77b0" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 33681 } } }
I20260926 23:22:00.619131 6174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:00.619204 6175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:00.620374 6181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:22:00.625851 6181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:22:00.634161 6181 catalog_manager.cc:1347] Generated new cluster ID: e7ec3028f1924707b82e01c9a7f388a6
I20260926 23:22:00.634282 6181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:22:00.661707 6181 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:22:00.663045 6181 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:22:00.677172 6181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17c66ba8730f41eea49e05e5247d77b0: Generated new TSK 0
I20260926 23:22:00.678305 6181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:22:00.691669 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6095
W20260926 23:22:00.692991 6158 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60919
W20260926 23:22:00.693671 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6095: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:22:00.693818 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 6095
2026-09-26T23:22:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:22:00 dist-test-slave-vcv1 krb5kdc[6194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:22:00 dist-test-slave-vcv1 krb5kdc[6194](info): set up 2 sockets
Sep 26 23:22:00 dist-test-slave-vcv1 krb5kdc[6194](info): commencing operation
krb5kdc: starting...
W20260926 23:22:01.805521 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:22:01 dist-test-slave-vcv1 krb5kdc[6194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:22:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:22:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:22:02.057035 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:46421
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:46421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:22:02.381593 6207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:02.381908 6207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:02.382061 6207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:02.389127 6207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:02.389261 6207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:02.389365 6207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:02.389468 6207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:02.389542 6207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:02.399194 6207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:46421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:46421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:02.400410 6207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:02.401664 6207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:22:02.413090 6213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:02.413157 6215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:02.413157 6212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:02.414228 6207 server_base.cc:1034] running on GCE node
I20260926 23:22:02.414886 6207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:02.416209 6207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:02.417413 6207 hybrid_clock.cc:648] HybridClock initialized: now 1790464922417374 us; error 30 us; skew 500 ppm
I20260926 23:22:02.417702 6207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:02 dist-test-slave-vcv1 krb5kdc[6194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464922, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:02.420796 6207 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:22:02.422204 6207 webserver.cc:460] Webserver started at http://127.16.62.126:45001/ using document root <none> and password file <none>
I20260926 23:22:02.422818 6207 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:02.422916 6207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:02.423164 6207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:02.426029 6207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "7d01dbcb993a449b9a6115edaec7025d"
format_stamp: "Formatted at 2026-09-26 23:22:02 on dist-test-slave-vcv1"
server_key: "f75c7096282c7da4e388fb0618ccb3be"
server_key_iv: "ad9d9d28d2d4a77509f963562caa3699"
server_key_version: "encryptionkey@0"
I20260926 23:22:02.426811 6207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "7d01dbcb993a449b9a6115edaec7025d"
format_stamp: "Formatted at 2026-09-26 23:22:02 on dist-test-slave-vcv1"
server_key: "f75c7096282c7da4e388fb0618ccb3be"
server_key_iv: "ad9d9d28d2d4a77509f963562caa3699"
server_key_version: "encryptionkey@0"
I20260926 23:22:02.431592 6207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260926 23:22:02.434878 6222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:02.436170 6207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:22:02.436338 6207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "7d01dbcb993a449b9a6115edaec7025d"
format_stamp: "Formatted at 2026-09-26 23:22:02 on dist-test-slave-vcv1"
server_key: "f75c7096282c7da4e388fb0618ccb3be"
server_key_iv: "ad9d9d28d2d4a77509f963562caa3699"
server_key_version: "encryptionkey@0"
I20260926 23:22:02.436511 6207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:02.458578 6207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:02.462832 6207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:02.463054 6207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:02.481384 6207 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:46421
I20260926 23:22:02.481417 6273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:46421 every 8 connection(s)
I20260926 23:22:02.482668 6207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:22:02.486145 6274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:02.489446 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 6207
I20260926 23:22:02.489603 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:22:02.490034 16633 external_mini_cluster.cc:1428] Setting key dd765abc0206578ec9a2d12c32e69994
I20260926 23:22:02.498047 6274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d: Bootstrap starting.
I20260926 23:22:02.501742 6274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:02.503022 6274 log.cc:824] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:22:02 dist-test-slave-vcv1 krb5kdc[6194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:22:02.505781 6274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d: No bootstrap required, opened a new log
I20260926 23:22:02.512897 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:02.491680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52567 (local address 127.16.62.126:46421)
0926 23:22:02.492439 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:02.492450 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:02.492487 (+ 37us) server_negotiation.cc:407] Connection header received
0926 23:22:02.494079 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:02.494099 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:02.494476 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:02.494785 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:02.496594 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:02.497865 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:02.499675 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:02.500395 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:02.500521 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:02.505366 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:02.505373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:02.505389 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:02.505426 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:02.509028 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:02.509666 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:02.509679 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:02.509684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:02.509846 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:02.510238 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:02.510243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:02.510246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:02.510751 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:02.511048 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:02.511469 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 23:22:02.511873 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":383,"threads_started":1}
I20260926 23:22:02.513952 6274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d01dbcb993a449b9a6115edaec7025d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46421 } }
I20260926 23:22:02.514307 6274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:22:02.514417 6274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d01dbcb993a449b9a6115edaec7025d, State: Initialized, Role: FOLLOWER
I20260926 23:22:02.515309 6274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d01dbcb993a449b9a6115edaec7025d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46421 } }
I20260926 23:22:02.515595 6279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:22:02.515851 6279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d01dbcb993a449b9a6115edaec7025d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46421 } }
I20260926 23:22:02.516479 6274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:22:02.516635 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:22:02.516510 6279 leader_election.cc:302] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d01dbcb993a449b9a6115edaec7025d; no voters:
I20260926 23:22:02.516669 6274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:22:02.516776 6274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:02.517316 6279 leader_election.cc:288] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260926 23:22:02.520040 6274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d01dbcb993a449b9a6115edaec7025d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46421 } }
I20260926 23:22:02.520383 6274 leader_election.cc:302] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d01dbcb993a449b9a6115edaec7025d; no voters:
I20260926 23:22:02.520520 6280 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260926 23:22:02.520711 6274 leader_election.cc:288] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:22:02.520880 6279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:22:02.522343 6279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [term 1 LEADER]: Becoming Leader. State: Replica: 7d01dbcb993a449b9a6115edaec7025d, State: Running, Role: LEADER
I20260926 23:22:02.522957 6279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d01dbcb993a449b9a6115edaec7025d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46421 } }
I20260926 23:22:02.524852 6274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:22:02.525121 6287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d01dbcb993a449b9a6115edaec7025d. Latest consensus state: current_term: 1 leader_uuid: "7d01dbcb993a449b9a6115edaec7025d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d01dbcb993a449b9a6115edaec7025d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46421 } } }
I20260926 23:22:02.525478 6287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:02.525054 6280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d01dbcb993a449b9a6115edaec7025d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d01dbcb993a449b9a6115edaec7025d" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 46421 } } }
I20260926 23:22:02.526078 6280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:02.526670 6289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:22:02.544754 6289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:22:02.553894 6289 catalog_manager.cc:1347] Generated new cluster ID: 842b603ce96c4d8b8d2ddbc887f619bb
I20260926 23:22:02.554013 6289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:22:02.565497 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:02.533786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42307 (local address 127.16.62.126:46421)
0926 23:22:02.534001 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:02.534010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:02.544091 (+ 10081us) server_negotiation.cc:407] Connection header received
0926 23:22:02.544700 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:02.544709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:02.544821 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:02.544915 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:02.547564 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:02.549113 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:02.551588 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:02.552581 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:02.552717 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:02.556821 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:02.556831 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:02.556839 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:02.556904 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:02.561434 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:02.562256 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:02.562263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:02.562267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:02.562450 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:02.562921 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:02.562927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:02.562930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:02.563309 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:02.563538 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:02.564849 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0926 23:22:02.565073 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:22:02.577721 6289 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:22:02.579039 6289 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:22:02.589668 6289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d01dbcb993a449b9a6115edaec7025d: Generated new TSK 0
I20260926 23:22:02.590443 6289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:22:02.596733 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6207
I20260926 23:22:02.598568 6269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:22:02.711119 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 6207
2026-09-26T23:22:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:22:02 dist-test-slave-vcv1 krb5kdc[6308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:22:02 dist-test-slave-vcv1 krb5kdc[6308](info): set up 2 sockets
Sep 26 23:22:02 dist-test-slave-vcv1 krb5kdc[6308](info): commencing operation
krb5kdc: starting...
W20260926 23:22:03.829651 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:22:03 dist-test-slave-vcv1 krb5kdc[6308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:22:04.081096 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:34629
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:33771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:34629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:22:04.399111 6321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:04.399394 6321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:04.399520 6321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:04.406800 6321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:04.406909 6321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:04.406962 6321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:04.407053 6321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:04.407135 6321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:04.416833 6321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:33771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:34629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:04.418102 6321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:04.419407 6321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:22:04.430606 6326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:04.430696 6327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:04.430868 6329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:04.431613 6321 server_base.cc:1034] running on GCE node
I20260926 23:22:04.432466 6321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:04.433796 6321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:04.435050 6321 hybrid_clock.cc:648] HybridClock initialized: now 1790464924435006 us; error 51 us; skew 500 ppm
I20260926 23:22:04.435398 6321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:04 dist-test-slave-vcv1 krb5kdc[6308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464924, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:04.438879 6321 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:22:04.440303 6321 webserver.cc:460] Webserver started at http://127.16.62.126:42325/ using document root <none> and password file <none>
I20260926 23:22:04.440945 6321 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:04.441090 6321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:04.441388 6321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:04.444350 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "50150ffd8616424db6c7b6cb32313910"
format_stamp: "Formatted at 2026-09-26 23:22:04 on dist-test-slave-vcv1"
server_key: "2924f95c1ccdf289a79ce0b3313ad883"
server_key_iv: "374437257aeead56f2fa1b51bfa32c10"
server_key_version: "encryptionkey@0"
I20260926 23:22:04.445098 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "50150ffd8616424db6c7b6cb32313910"
format_stamp: "Formatted at 2026-09-26 23:22:04 on dist-test-slave-vcv1"
server_key: "2924f95c1ccdf289a79ce0b3313ad883"
server_key_iv: "374437257aeead56f2fa1b51bfa32c10"
server_key_version: "encryptionkey@0"
I20260926 23:22:04.450078 6321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:22:04.453356 6336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:04.454586 6321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:22:04.454770 6321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "50150ffd8616424db6c7b6cb32313910"
format_stamp: "Formatted at 2026-09-26 23:22:04 on dist-test-slave-vcv1"
server_key: "2924f95c1ccdf289a79ce0b3313ad883"
server_key_iv: "374437257aeead56f2fa1b51bfa32c10"
server_key_version: "encryptionkey@0"
I20260926 23:22:04.454954 6321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:04.480017 6321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:04.484376 6321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:04.484663 6321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:04.501453 6321 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:34629
I20260926 23:22:04.501473 6387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:34629 every 8 connection(s)
I20260926 23:22:04.502739 6321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:22:04.503499 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 6321
I20260926 23:22:04.503684 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:22:04.504297 16633 external_mini_cluster.cc:1428] Setting key 030ed37636e7d8a38db6ca991b10f2a9
I20260926 23:22:04.507019 6388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:04.516379 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:04.506022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57271 (local address 127.16.62.126:34629)
0926 23:22:04.506761 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:04.506772 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:04.506811 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:22:04.508473 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:04.508494 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:04.508835 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:04.509252 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:04.509794 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:04.511006 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:04.512323 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:04.512935 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:04.513126 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:04.514227 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:04.514234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:04.514249 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:22:04.514778 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:04.514855 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:22:04.514877 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:04.515009 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:04.515207 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 23:22:04.515444 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":333,"threads_started":1}
I20260926 23:22:04.517273 6388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910: Bootstrap starting.
I20260926 23:22:04.519904 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:22:04.520815 6388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:04.521757 6388 log.cc:824] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910: Log is configured to *not* fsync() on all Append() calls
I20260926 23:22:04.524703 6388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910: No bootstrap required, opened a new log
I20260926 23:22:04.530872 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:04.524453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41471 (local address 127.16.62.126:34629)
0926 23:22:04.524608 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:04.524614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:04.525185 (+ 571us) server_negotiation.cc:407] Connection header received
0926 23:22:04.525316 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:04.525322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:04.525401 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:04.525514 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:04.525938 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:04.527069 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:04.528379 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:04.529057 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:04.529176 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:04.530133 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:04.530144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:04.530149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:22:04.530169 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:04.530217 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:22:04.530236 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:04.530338 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:04.530437 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 23:22:04.530523 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 23:22:04.533720 6388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50150ffd8616424db6c7b6cb32313910" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34629 } }
I20260926 23:22:04.534175 6388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:22:04.534274 6388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50150ffd8616424db6c7b6cb32313910, State: Initialized, Role: FOLLOWER
I20260926 23:22:04.534801 6388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50150ffd8616424db6c7b6cb32313910" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34629 } }
I20260926 23:22:04.535025 6388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:22:04.535164 6388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:22:04.535338 6388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:04.537515 6388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50150ffd8616424db6c7b6cb32313910" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34629 } }
I20260926 23:22:04.537981 6388 leader_election.cc:302] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50150ffd8616424db6c7b6cb32313910; no voters:
I20260926 23:22:04.538594 6388 leader_election.cc:288] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:22:04.538753 6398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:22:04.540596 6388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:22:04.541043 6398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [term 1 LEADER]: Becoming Leader. State: Replica: 50150ffd8616424db6c7b6cb32313910, State: Running, Role: LEADER
I20260926 23:22:04.541548 6398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50150ffd8616424db6c7b6cb32313910" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34629 } }
I20260926 23:22:04.544634 6399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50150ffd8616424db6c7b6cb32313910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50150ffd8616424db6c7b6cb32313910" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34629 } } }
I20260926 23:22:04.545074 6399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:04.544622 6400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50150ffd8616424db6c7b6cb32313910. Latest consensus state: current_term: 1 leader_uuid: "50150ffd8616424db6c7b6cb32313910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50150ffd8616424db6c7b6cb32313910" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34629 } } }
I20260926 23:22:04.545444 6400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910 [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:04.546955 6407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:22:04.552205 6407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:22:04.560428 6407 catalog_manager.cc:1347] Generated new cluster ID: ea88beaa09cc4f9d8265c45b4c15a821
I20260926 23:22:04.560544 6407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:22:04.582017 6407 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:22:04.583243 6407 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:22:04.593986 6407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50150ffd8616424db6c7b6cb32313910: Generated new TSK 0
I20260926 23:22:04.594745 6407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:22:04.602801 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6321
W20260926 23:22:04.603852 6383 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57271
W20260926 23:22:04.604527 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6321: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:22:04.604671 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 6321
2026-09-26T23:22:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:22:04 dist-test-slave-vcv1 krb5kdc[6419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:22:04 dist-test-slave-vcv1 krb5kdc[6419](info): set up 2 sockets
Sep 26 23:22:04 dist-test-slave-vcv1 krb5kdc[6419](info): commencing operation
krb5kdc: starting...
W20260926 23:22:05.689996 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:22:05 dist-test-slave-vcv1 krb5kdc[6419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:22:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:22:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:22:05.930994 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:34073
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:34073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:22:06.251464 6432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:06.251754 6432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:06.251847 6432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:06.259778 6432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:06.259893 6432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:06.259949 6432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:06.260010 6432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:06.260054 6432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:06.269932 6432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:34073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:34073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:06.271195 6432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:06.272505 6432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:22:06.284601 6440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:06.284564 6438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:06.284644 6437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:06.289609 6432 server_base.cc:1034] running on GCE node
I20260926 23:22:06.290447 6432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:06.291920 6432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:06.293116 6432 hybrid_clock.cc:648] HybridClock initialized: now 1790464926293045 us; error 60 us; skew 500 ppm
I20260926 23:22:06.293412 6432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:06 dist-test-slave-vcv1 krb5kdc[6419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464926, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:06.307631 6432 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:22:06.309316 6432 webserver.cc:460] Webserver started at http://127.16.62.126:35329/ using document root <none> and password file <none>
I20260926 23:22:06.310312 6432 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:06.310473 6432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:06.310834 6432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:06.313872 6432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "4811ce9d44df442686fc3b54c1662e0a"
format_stamp: "Formatted at 2026-09-26 23:22:06 on dist-test-slave-vcv1"
server_key: "d7c85950a3d706036713cfb128266b3e"
server_key_iv: "abab07641e305c90800ea31dba8fe565"
server_key_version: "encryptionkey@0"
I20260926 23:22:06.314563 6432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "4811ce9d44df442686fc3b54c1662e0a"
format_stamp: "Formatted at 2026-09-26 23:22:06 on dist-test-slave-vcv1"
server_key: "d7c85950a3d706036713cfb128266b3e"
server_key_iv: "abab07641e305c90800ea31dba8fe565"
server_key_version: "encryptionkey@0"
I20260926 23:22:06.319574 6432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:22:06.322893 6448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:06.324249 6432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:22:06.324415 6432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "4811ce9d44df442686fc3b54c1662e0a"
format_stamp: "Formatted at 2026-09-26 23:22:06 on dist-test-slave-vcv1"
server_key: "d7c85950a3d706036713cfb128266b3e"
server_key_iv: "abab07641e305c90800ea31dba8fe565"
server_key_version: "encryptionkey@0"
I20260926 23:22:06.324612 6432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:06.330044 6432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:06.334167 6432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:06.334452 6432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:06.352090 6432 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:34073
I20260926 23:22:06.352128 6499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:34073 every 8 connection(s)
I20260926 23:22:06.353590 6432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:22:06.357175 6501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:06.362721 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 6432
I20260926 23:22:06.362890 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:22:06.363319 16633 external_mini_cluster.cc:1428] Setting key fde2737a89fd2c294d39e59b020c4114
I20260926 23:22:06.367657 6501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a: Bootstrap starting.
I20260926 23:22:06.370648 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:06.364978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45843 (local address 127.16.62.126:34073)
0926 23:22:06.365842 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:06.365855 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:06.365895 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:22:06.367486 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:06.367506 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:06.367851 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:06.368225 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:06.369202 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:06.369208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:06.369222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:22:06.369235 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:06.369296 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:22:06.369315 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:06.369475 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:06.369599 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 23:22:06.369775 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":436,"threads_started":1}
I20260926 23:22:06.371271 6501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:06.372444 6501 log.cc:824] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:22:06.374253 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:22:06.375123 6501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a: No bootstrap required, opened a new log
I20260926 23:22:06.380542 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:06.378325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37107 (local address 127.16.62.126:34073)
0926 23:22:06.378460 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:06.378464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:06.378798 (+ 334us) server_negotiation.cc:407] Connection header received
0926 23:22:06.378943 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:06.378946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:06.378986 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:06.379058 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:06.379927 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:06.379931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:06.379934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:22:06.379947 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:06.379979 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:22:06.379996 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:06.380084 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:06.380221 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 23:22:06.380264 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 23:22:06.383307 6501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4811ce9d44df442686fc3b54c1662e0a" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34073 } }
I20260926 23:22:06.383685 6501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:22:06.383786 6501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4811ce9d44df442686fc3b54c1662e0a, State: Initialized, Role: FOLLOWER
I20260926 23:22:06.384371 6501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4811ce9d44df442686fc3b54c1662e0a" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34073 } }
I20260926 23:22:06.384595 6501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:22:06.384740 6501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:22:06.384912 6501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:06.387042 6501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4811ce9d44df442686fc3b54c1662e0a" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34073 } }
I20260926 23:22:06.387490 6501 leader_election.cc:302] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4811ce9d44df442686fc3b54c1662e0a; no voters:
I20260926 23:22:06.388034 6501 leader_election.cc:288] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:22:06.388173 6511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:22:06.388830 6511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [term 1 LEADER]: Becoming Leader. State: Replica: 4811ce9d44df442686fc3b54c1662e0a, State: Running, Role: LEADER
I20260926 23:22:06.389418 6511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4811ce9d44df442686fc3b54c1662e0a" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34073 } }
I20260926 23:22:06.390165 6501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:22:06.392068 6513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4811ce9d44df442686fc3b54c1662e0a. Latest consensus state: current_term: 1 leader_uuid: "4811ce9d44df442686fc3b54c1662e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4811ce9d44df442686fc3b54c1662e0a" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34073 } } }
I20260926 23:22:06.392552 6513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:06.392079 6512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4811ce9d44df442686fc3b54c1662e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4811ce9d44df442686fc3b54c1662e0a" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 34073 } } }
I20260926 23:22:06.392928 6512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:06.394101 6516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:22:06.398876 6516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:22:06.408668 6516 catalog_manager.cc:1347] Generated new cluster ID: 75c210c06f474ef6a498d7e5432c34e0
I20260926 23:22:06.408767 6516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:22:06.461231 6516 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:22:06.461669 6516 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:22:06.468770 6516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4811ce9d44df442686fc3b54c1662e0a: Generated new TSK 0
I20260926 23:22:06.469555 6516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:22:06.489279 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6432
W20260926 23:22:06.490336 6495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45843
W20260926 23:22:06.491003 16633 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6432: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:22:06.491165 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 6432
2026-09-26T23:22:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1889 ms)
[----------] 18 tests from AuthTokenIssuingTest (34421 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:22:06 dist-test-slave-vcv1 krb5kdc[6532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:22:06 dist-test-slave-vcv1 krb5kdc[6532](info): set up 2 sockets
Sep 26 23:22:06 dist-test-slave-vcv1 krb5kdc[6532](info): commencing operation
krb5kdc: starting...
W20260926 23:22:07.593667 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:22:07 dist-test-slave-vcv1 krb5kdc[6532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:22:07.843212 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:40221
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:40221,127.16.62.125:46611,127.16.62.124:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:22:08.164494 6545 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:22:08.164826 6545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:08.164938 6545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:08.165066 6545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:08.172590 6545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:08.172660 6545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:08.172750 6545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:08.172814 6545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:08.172897 6545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:08.182499 6545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:40221,127.16.62.125:46611,127.16.62.124:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:40221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:08.183792 6545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:08.185143 6545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:22:08.196667 6550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:08.196619 6551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:08.196580 6553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:08.197608 6545 server_base.cc:1034] running on GCE node
I20260926 23:22:08.198382 6545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:08.199717 6545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:08.200946 6545 hybrid_clock.cc:648] HybridClock initialized: now 1790464928200889 us; error 37 us; skew 500 ppm
I20260926 23:22:08.201256 6545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:08 dist-test-slave-vcv1 krb5kdc[6532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464928, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:08.204356 6545 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:22:08.205770 6545 webserver.cc:460] Webserver started at http://127.16.62.126:41073/ using document root <none> and password file <none>
I20260926 23:22:08.206418 6545 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:08.206542 6545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:08.206849 6545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:08.209884 6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "df86c33d09d3491bb378675636f601fa"
format_stamp: "Formatted at 2026-09-26 23:22:08 on dist-test-slave-vcv1"
server_key: "09e26d4710896e45087ad136861bdc1f"
server_key_iv: "b94dbe0448bd35d02ed55e5fdcede32d"
server_key_version: "encryptionkey@0"
I20260926 23:22:08.210641 6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "df86c33d09d3491bb378675636f601fa"
format_stamp: "Formatted at 2026-09-26 23:22:08 on dist-test-slave-vcv1"
server_key: "09e26d4710896e45087ad136861bdc1f"
server_key_iv: "b94dbe0448bd35d02ed55e5fdcede32d"
server_key_version: "encryptionkey@0"
I20260926 23:22:08.215526 6545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 23:22:08.218878 6560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:08.220111 6545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:22:08.220340 6545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "df86c33d09d3491bb378675636f601fa"
format_stamp: "Formatted at 2026-09-26 23:22:08 on dist-test-slave-vcv1"
server_key: "09e26d4710896e45087ad136861bdc1f"
server_key_iv: "b94dbe0448bd35d02ed55e5fdcede32d"
server_key_version: "encryptionkey@0"
I20260926 23:22:08.220567 6545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:08.242373 6545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:08.246778 6545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:08.247077 6545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:08.266126 6545 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:40221
I20260926 23:22:08.266170 6612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:40221 every 8 connection(s)
I20260926 23:22:08.267493 6545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:22:08.271289 6613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:08.276521 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 6545
I20260926 23:22:08.276721 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:22:08.277357 16633 external_mini_cluster.cc:1428] Setting key 23c8476d3aa3446f2250fb1cac31f635
I20260926 23:22:08.278033 6613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:08 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464928, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:22:08.300951 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.281310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:40221 (local address 127.0.0.1:40726)
0926 23:22:08.282261 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.282311 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:08.285184 (+ 2873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:08.286070 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:08.286075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:08.286631 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:08.286953 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:08.286969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.288651 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.288658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:08.289703 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:08.289719 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.290604 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.290611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:08.290784 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:08.291442 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:08.291468 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:08.293833 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:08.296965 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:08.296977 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:08.296989 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:08.297355 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:08.297745 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:08.297752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:08.297755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:08.297901 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:08.298585 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:08.298598 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:08.298799 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 23:22:08.299031 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 23:22:08.299587 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":128,"thread_start_us":406,"threads_started":1}
I20260926 23:22:08.300917 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.281310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.16.62.126:40221)
0926 23:22:08.283211 (+ 1901us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:08.283217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:08.283260 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:22:08.285579 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:08.285585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:08.285814 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:08.286008 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:08.287177 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.288508 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.289844 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.290379 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.290566 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:08.294044 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:08.294052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:08.294063 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:08.294095 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:08.296839 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:08.297493 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:08.297499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:08.297501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:08.297645 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:08.297999 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:08.298006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:08.298012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:08.298459 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:08.298676 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:08.299142 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 23:22:08.299465 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1712,"spinlock_wait_cycles":42880,"thread_start_us":1493,"threads_started":1}
WARNING: no policy specified for kudu/127.16.62.125@KRBTEST.COM; defaulting to no policy
I20260926 23:22:08.303864 6613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.125@KRBTEST.COM" created.
I20260926 23:22:08.305147 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.304709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50258)
0926 23:22:08.304911 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.305036 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 23:22:08.305411 6562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
I20260926 23:22:08.306849 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.306518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50268)
0926 23:22:08.306671 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.306732 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:22:08.307054 6613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:46611: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab.
Entry for principal kudu/127.16.62.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab.
I20260926 23:22:08.335384 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.125:46611
--webserver_interface=127.16.62.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab
--principal=kudu/127.16.62.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:40221,127.16.62.125:46611,127.16.62.124:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:22:08.356351 6613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } attempt: 1
I20260926 23:22:08.357592 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.357121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50282)
0926 23:22:08.357329 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.357455 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 23:22:08.358675 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.358360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50292)
0926 23:22:08.358489 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.358553 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 23:22:08.358965 6613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:46611: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
I20260926 23:22:08.427222 6613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } attempt: 2
I20260926 23:22:08.428439 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.427941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50308)
0926 23:22:08.428217 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.428305 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 23:22:08.429409 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.429061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50316)
0926 23:22:08.429215 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.429291 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 23:22:08.429591 6613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:46611: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
I20260926 23:22:08.520833 6613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } attempt: 3
I20260926 23:22:08.521946 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.521530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50328)
0926 23:22:08.521724 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.521817 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 23:22:08.522890 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.522567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50342)
0926 23:22:08.522711 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.522771 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 23:22:08.523057 6613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:46611: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
W20260926 23:22:08.653434 6620 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:22:08.653810 6620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:08.653929 6620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:08.654058 6620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:08.661134 6620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:08.661242 6620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:08.661352 6620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:08.661437 6620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:08.661538 6620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:08.666285 6613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } attempt: 4
I20260926 23:22:08.667477 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.666884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50346)
0926 23:22:08.667217 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.667299 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260926 23:22:08.668444 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.668087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50352)
0926 23:22:08.668255 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.668336 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 23:22:08.668648 6613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:46611: Network error: Client connection negotiation failed: client connection to 127.16.62.125:46611: connect: Connection refused (error 111)
I20260926 23:22:08.671541 6620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:40221,127.16.62.125:46611,127.16.62.124:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab
--principal=kudu/127.16.62.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.125:46611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.62.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:08.672796 6620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:08.674036 6620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:22:08.685276 6625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:08.685329 6628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:08.685243 6626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:08.686411 6620 server_base.cc:1034] running on GCE node
I20260926 23:22:08.687120 6620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:08.688455 6620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:08.689707 6620 hybrid_clock.cc:648] HybridClock initialized: now 1790464928689648 us; error 33 us; skew 500 ppm
I20260926 23:22:08.690069 6620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:08 dist-test-slave-vcv1 krb5kdc[6532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464928, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:08.693234 6620 init.cc:377] Logged in from keytab as kudu/127.16.62.125@KRBTEST.COM (short username kudu)
I20260926 23:22:08.694564 6620 webserver.cc:460] Webserver started at http://127.16.62.125:36611/ using document root <none> and password file <none>
I20260926 23:22:08.695226 6620 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:08.695349 6620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:08.695667 6620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:08.698529 6620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/data/instance:
uuid: "7d9b9da93637494ca63c68b7b328eda6"
format_stamp: "Formatted at 2026-09-26 23:22:08 on dist-test-slave-vcv1"
server_key: "2910b64ee78f4190317772b21e76d4e1"
server_key_iv: "8e9a4e6af0bdb9ed33655bf28e8c2529"
server_key_version: "encryptionkey@0"
I20260926 23:22:08.699291 6620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/wal/instance:
uuid: "7d9b9da93637494ca63c68b7b328eda6"
format_stamp: "Formatted at 2026-09-26 23:22:08 on dist-test-slave-vcv1"
server_key: "2910b64ee78f4190317772b21e76d4e1"
server_key_iv: "8e9a4e6af0bdb9ed33655bf28e8c2529"
server_key_version: "encryptionkey@0"
I20260926 23:22:08.704123 6620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 23:22:08.707299 6635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:08.708671 6620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 23:22:08.708817 6620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/wal
uuid: "7d9b9da93637494ca63c68b7b328eda6"
format_stamp: "Formatted at 2026-09-26 23:22:08 on dist-test-slave-vcv1"
server_key: "2910b64ee78f4190317772b21e76d4e1"
server_key_iv: "8e9a4e6af0bdb9ed33655bf28e8c2529"
server_key_version: "encryptionkey@0"
I20260926 23:22:08.708997 6620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:08.723313 6620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:08.727494 6620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:08.727761 6620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:08.745146 6620 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.125:46611
I20260926 23:22:08.745165 6686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.125:46611 every 8 connection(s)
I20260926 23:22:08.746446 6620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
I20260926 23:22:08.748430 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 6620
I20260926 23:22:08.748608 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-1/wal/instance
I20260926 23:22:08.749056 16633 external_mini_cluster.cc:1428] Setting key 033a9c64cda56bba1b5d5898345cfecb
I20260926 23:22:08.750000 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:08.759227 6687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:08 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464928, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.62.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.124@KRBTEST.COM" created.
I20260926 23:22:08.780588 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.762415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.16.62.126:40221)
0926 23:22:08.762666 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:08.762672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:08.763888 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 23:22:08.765806 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:08.765815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:08.765904 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:08.766016 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:08.767396 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.768537 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.769948 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.770730 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.770875 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:08.774671 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:08.774676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:08.774679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:08.774711 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:08.777371 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:08.778153 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:08.778161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:08.778168 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:08.778311 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:08.778698 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:08.778706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:08.778712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:08.779105 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:08.779280 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:08.779881 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 23:22:08.780168 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:22:08.781147 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.762925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:40221 (local address 127.0.0.1:40728)
0926 23:22:08.763710 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.763757 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:08.765529 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:08.766204 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:08.766211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:08.766702 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:08.767221 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:08.767237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.768679 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.768683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:08.769809 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:08.769821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.770925 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.770929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:08.771114 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:08.771769 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:08.771790 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:08.774442 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:08.777489 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:08.777499 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:08.777516 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:08.778026 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:08.778419 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:08.778422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:08.778424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:08.778572 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:08.779248 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:08.779253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:08.779515 (+ 262us) client_negotiation.cc:769] Sending connection context
0926 23:22:08.779747 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 23:22:08.780069 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":330,"threads_started":1}
I20260926 23:22:08.782729 6687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:08 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464928, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:22:08.797406 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.783647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.16.62.125:46611)
0926 23:22:08.784254 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:08.784260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:08.784283 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:22:08.784322 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:08.784327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:08.784479 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:08.784635 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:08.785312 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.786320 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.787386 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.788320 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.788458 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:08.791792 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:08.791802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:08.791814 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:08.791840 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:08.794501 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:08.795306 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:08.795312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:08.795314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:08.795417 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:08.795830 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:08.795833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:08.795835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:08.796204 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:08.796349 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:08.796797 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 23:22:08.797111 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":406,"thread_start_us":266,"threads_started":1}
I20260926 23:22:08.797516 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.783539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50366)
0926 23:22:08.783702 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.783724 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:08.783820 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:08.784609 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:08.784613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:08.785025 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:08.785147 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:08.785154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.786443 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.786449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:08.787262 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:08.787272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.788472 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.788476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:08.788653 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:08.789202 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:08.789218 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:08.791568 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:08.794632 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:08.794639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:08.794646 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:08.795184 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:08.795525 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:08.795532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:08.795539 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:08.795732 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:08.796336 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:08.796344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:08.796508 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 23:22:08.796678 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 23:22:08.797009 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":29}
I20260926 23:22:08.799780 6687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:22:08.800796 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.800407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52210)
0926 23:22:08.800561 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.800667 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:22:08.801096 6637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
I20260926 23:22:08.802538 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.802259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52218)
0926 23:22:08.802382 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.802451 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:22:08.802798 6687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:45983: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab.
Entry for principal kudu/127.16.62.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab.
I20260926 23:22:08.809160 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.124:45983
--webserver_interface=127.16.62.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab
--principal=kudu/127.16.62.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:40221,127.16.62.125:46611,127.16.62.124:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:22:08.852118 6687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } attempt: 1
I20260926 23:22:08.853343 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.852917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52234)
0926 23:22:08.853106 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.853206 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 23:22:08.854379 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.854011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52238)
0926 23:22:08.854205 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.854276 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 23:22:08.854593 6687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:45983: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
I20260926 23:22:08.922793 6687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } attempt: 2
I20260926 23:22:08.924033 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.923476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52254)
0926 23:22:08.923709 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.923837 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 23:22:08.925220 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.924844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52266)
0926 23:22:08.924997 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.925083 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:22:08.925424 6687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:45983: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
I20260926 23:22:08.967941 6613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } attempt: 5
Sep 26 23:22:08 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464928, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:22:08.982141 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.968877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.16.62.125:46611)
0926 23:22:08.969073 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:08.969079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:08.969102 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:22:08.969282 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:08.969286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:08.969350 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:08.969455 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:08.970232 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.970931 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.972095 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.972587 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.972716 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:08.976133 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:08.976182 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:08.976187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:08.976227 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:08.979209 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:08.980088 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:08.980091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:08.980094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:08.980279 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:08.980772 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:08.980775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:08.980777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:08.981060 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:08.981163 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:08.981729 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 23:22:08.981891 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:22:08.982254 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.968740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50372)
0926 23:22:08.968932 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.968954 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:08.969121 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:08.969494 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:08.969498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:08.969903 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:08.970086 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:08.970095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.971075 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.971082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:08.971937 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:08.971947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:08.972729 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:08.972735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:08.972950 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:08.973664 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:08.973681 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:08.975942 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:08.979344 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:08.979351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:08.979356 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:08.979967 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:08.980414 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:08.980422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:08.980427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:08.980636 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:08.981205 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:08.981212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:08.981424 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 23:22:08.981620 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 23:22:08.981918 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 23:22:08.983390 6613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:22:08.984475 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.984015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52282)
0926 23:22:08.984213 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.984303 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 23:22:08.985406 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:08.985050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52288)
0926 23:22:08.985210 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:08.985296 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 23:22:08.985648 6613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:45983: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
I20260926 23:22:09.016664 6687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } attempt: 3
I20260926 23:22:09.017830 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.017379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52296)
0926 23:22:09.017577 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.017675 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 23:22:09.018954 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.018536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52302)
0926 23:22:09.018754 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.018831 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260926 23:22:09.019194 6687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:45983: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
I20260926 23:22:09.036870 6613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } attempt: 1
I20260926 23:22:09.038105 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.037543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52304)
0926 23:22:09.037774 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.037894 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 23:22:09.039139 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.038772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52310)
0926 23:22:09.038917 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.038999 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 23:22:09.039367 6613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:45983: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
I20260926 23:22:09.107625 6613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } attempt: 2
I20260926 23:22:09.108915 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.108355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52318)
0926 23:22:09.108587 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.108706 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 23:22:09.109851 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.109524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52324)
0926 23:22:09.109674 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.109748 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 23:22:09.110188 6613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:45983: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
W20260926 23:22:09.139343 6694 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:22:09.139642 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:09.139714 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:09.139796 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:09.147929 6694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:09.148038 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:09.148098 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:09.148208 6694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:09.148283 6694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:09.158133 6694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:40221,127.16.62.125:46611,127.16.62.124:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab
--principal=kudu/127.16.62.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.124:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.62.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:09.159271 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:09.160624 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:22:09.162385 6687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } attempt: 4
I20260926 23:22:09.163520 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.163078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52338)
0926 23:22:09.163284 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.163368 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 23:22:09.164518 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.164202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52354)
0926 23:22:09.164345 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.164416 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 23:22:09.164752 6687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:45983: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
W20260926 23:22:09.172350 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:09.172498 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:09.172521 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:09.173471 6694 server_base.cc:1034] running on GCE node
I20260926 23:22:09.174196 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:09.175503 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:09.176712 6694 hybrid_clock.cc:648] HybridClock initialized: now 1790464929176653 us; error 40 us; skew 500 ppm
I20260926 23:22:09.177033 6694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:09 dist-test-slave-vcv1 krb5kdc[6532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464929, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:09.179970 6694 init.cc:377] Logged in from keytab as kudu/127.16.62.124@KRBTEST.COM (short username kudu)
I20260926 23:22:09.181326 6694 webserver.cc:460] Webserver started at http://127.16.62.124:36175/ using document root <none> and password file <none>
I20260926 23:22:09.181993 6694 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:09.182130 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:09.182423 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:09.185309 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/data/instance:
uuid: "cca7a0c536fc49318e422a699ec8e019"
format_stamp: "Formatted at 2026-09-26 23:22:09 on dist-test-slave-vcv1"
server_key: "60e0ffeaf97b7fa5420a03090c11093f"
server_key_iv: "948b90e147651648017248950e8c0d81"
server_key_version: "encryptionkey@0"
I20260926 23:22:09.186070 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/wal/instance:
uuid: "cca7a0c536fc49318e422a699ec8e019"
format_stamp: "Formatted at 2026-09-26 23:22:09 on dist-test-slave-vcv1"
server_key: "60e0ffeaf97b7fa5420a03090c11093f"
server_key_iv: "948b90e147651648017248950e8c0d81"
server_key_version: "encryptionkey@0"
I20260926 23:22:09.190940 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:22:09.194199 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:09.195441 6694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:22:09.195616 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/wal
uuid: "cca7a0c536fc49318e422a699ec8e019"
format_stamp: "Formatted at 2026-09-26 23:22:09 on dist-test-slave-vcv1"
server_key: "60e0ffeaf97b7fa5420a03090c11093f"
server_key_iv: "948b90e147651648017248950e8c0d81"
server_key_version: "encryptionkey@0"
I20260926 23:22:09.195829 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:09.216467 6613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } attempt: 3
I20260926 23:22:09.217482 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.217071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52368)
0926 23:22:09.217261 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.217340 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 23:22:09.218465 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.218079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52374)
0926 23:22:09.218240 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.218332 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 23:22:09.218742 6613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:45983: Network error: Client connection negotiation failed: client connection to 127.16.62.124:45983: connect: Connection refused (error 111)
I20260926 23:22:09.221311 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:09.225279 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:09.225540 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:09.242810 6694 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.124:45983
I20260926 23:22:09.242816 6761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.124:45983 every 8 connection(s)
I20260926 23:22:09.244098 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
I20260926 23:22:09.247535 6762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:09.251843 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 6694
I20260926 23:22:09.252022 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464575313384-16633-0/minicluster-data/master-2/wal/instance
I20260926 23:22:09.252480 16633 external_mini_cluster.cc:1428] Setting key 4acad5c0d351558f68202923263b2315
I20260926 23:22:09.254186 6762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:09 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464929, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
Sep 26 23:22:09 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
I20260926 23:22:09.273384 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.257291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.16.62.126:40221)
0926 23:22:09.257562 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.257569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.258061 (+ 492us) server_negotiation.cc:407] Connection header received
0926 23:22:09.258279 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.258285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.258363 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.258487 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.259688 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.260438 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.262094 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.262619 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.262710 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.266659 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.266668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.266675 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.266714 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.269997 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.270860 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.270866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.270870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.271037 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.271445 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.271452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.271455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.271813 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.271986 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.272787 (+ 801us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.273009 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:22:09.274470 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.257187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:40221 (local address 127.0.0.1:40732)
0926 23:22:09.257896 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.257949 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:09.258122 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:09.258556 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:09.258562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:09.259175 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:09.259507 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.259523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.260581 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.260586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.261949 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.261960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.262818 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.262825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.263195 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.263965 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:09.263998 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:09.266442 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:09.270175 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.270187 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.270199 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.270747 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.271131 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.271136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.271140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.271335 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.271955 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:09.271963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:09.272361 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 23:22:09.272676 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 23:22:09.273155 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":103,"thread_start_us":347,"threads_started":1}
I20260926 23:22:09.274838 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.254220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.16.62.124:45983)
0926 23:22:09.255083 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.255100 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.255149 (+ 49us) server_negotiation.cc:407] Connection header received
0926 23:22:09.257355 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.257387 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.257929 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.258268 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.259662 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.261108 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.262354 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.263104 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.263224 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.267275 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.267284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.267298 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.267331 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.270930 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.271721 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.271731 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.271736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.271874 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.272340 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.272347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.272349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.272755 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.273029 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.274152 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.274378 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":638,"thread_start_us":377,"threads_started":1}
I20260926 23:22:09.276088 6762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:09 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464929, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
Sep 26 23:22:09 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:22:09.290205 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.276975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:46611 (local address 127.0.0.1:50376)
0926 23:22:09.277250 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.277268 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:09.277375 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:09.277683 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:09.277687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:09.278144 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:09.278355 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.278366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.279313 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.279318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.280378 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.280389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.281334 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.281337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.281490 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.282040 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:09.282077 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:09.284325 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:09.287874 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.287877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.287880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.288387 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.288765 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.288769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.288771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.288901 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.289552 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:09.289556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:09.289679 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 23:22:09.289780 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 23:22:09.289970 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260926 23:22:09.290529 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.277032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.16.62.125:46611)
0926 23:22:09.277207 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.277211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.277345 (+ 134us) server_negotiation.cc:407] Connection header received
0926 23:22:09.277485 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.277491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.277567 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.277663 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.278522 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.279180 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.280540 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.281204 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.281335 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.284528 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.284536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.284543 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.284589 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.287749 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.288496 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.288502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.288505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.288655 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.289036 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.289040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.289042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.289436 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.289589 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.289925 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.290143 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:22:09.291484 6762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:22:09.293361 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.278035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.16.62.126:40221)
0926 23:22:09.278179 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.278183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.278198 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:22:09.278240 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.278243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.278290 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.278364 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.279586 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.280420 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.282123 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.282770 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.282890 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.287258 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.287266 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.287273 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.287314 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.290679 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.291286 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.291290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.291292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.291447 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.291882 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.291888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.291890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.292254 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.292537 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.292830 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.293050 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 23:22:09 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464929, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
Sep 26 23:22:09 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:22:09.305217 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.292246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.16.62.124:45983)
0926 23:22:09.292385 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.292390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.292406 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:22:09.292547 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.292551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.292620 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.292723 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.293628 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.294548 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.295862 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.296505 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.296633 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.299957 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.299966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.299972 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.300012 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.302604 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.303333 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.303337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.303341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.303468 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.303798 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.303805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.303808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.304069 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.304230 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.304543 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.304812 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":46}
I20260926 23:22:09.305286 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.292170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52402)
0926 23:22:09.292329 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.292346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:09.292435 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:09.292760 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:09.292764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:09.293206 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:09.293373 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.293387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.294667 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.294671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.295716 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.295727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.296789 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.296794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.297007 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.297560 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:09.297575 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:09.299788 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:09.302743 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.302749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.302752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.303229 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.303576 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.303582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.303585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.303701 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.304202 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:09.304207 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:09.304339 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 23:22:09.304471 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 23:22:09.304906 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":148}
I20260926 23:22:09.309473 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.294986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.16.62.125:46611)
0926 23:22:09.295166 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.295174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.295189 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:22:09.295252 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.295255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.295303 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.295394 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.296973 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.297837 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.299469 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.299921 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.300035 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.304130 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.304177 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.304183 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.304221 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.306986 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.307600 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.307613 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.307617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.307749 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.308096 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.308103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.308106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.308408 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.308696 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.308949 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.309169 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:22:09.310001 6762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019: Bootstrap starting.
I20260926 23:22:09.313663 6762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:09.314034 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:22:09.314756 6762 log.cc:824] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019: Log is configured to *not* fsync() on all Append() calls
I20260926 23:22:09.317469 6762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019: No bootstrap required, opened a new log
I20260926 23:22:09.316973 6587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "df86c33d09d3491bb378675636f601fa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40746
I20260926 23:22:09.324368 6762 raft_consensus.cc:348] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } }
I20260926 23:22:09.324687 6762 raft_consensus.cc:729] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca7a0c536fc49318e422a699ec8e019, State: Initialized, Role: FOLLOWER
I20260926 23:22:09.325114 6762 consensus_queue.cc:260] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } }
I20260926 23:22:09.325989 6768 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.326372 6768 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:09.327040 6762 sys_catalog.cc:564] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:22:09.335670 6779 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:22:09.335805 6779 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:22:09.334182 6694 master_runner.cc:186] Error getting master registration for 127.16.62.126:40221: OK, instance_id { permanent_uuid: "df86c33d09d3491bb378675636f601fa" instance_seqno: 1790464928260591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:22:09.345383 6694 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:22:09.395982 6613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } attempt: 4
Sep 26 23:22:09 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464928, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
I20260926 23:22:09.411422 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.396747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52408)
0926 23:22:09.396994 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.397017 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:09.397166 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:09.397606 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:09.397611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:09.398121 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:09.398331 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.398342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.399806 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.399812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.401367 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.401383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.402375 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.402381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.402597 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.403268 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:09.403290 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:09.405899 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:09.409013 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.409019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.409023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.409368 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.409710 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.409716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.409720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.409865 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.410421 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:09.410426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:09.410590 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 23:22:09.410752 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 23:22:09.411023 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 23:22:09.411437 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.396822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.16.62.124:45983)
0926 23:22:09.397022 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.397027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.397082 (+ 55us) server_negotiation.cc:407] Connection header received
0926 23:22:09.397310 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.397315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.397422 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.397533 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.398474 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.399626 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.401512 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.402228 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.402344 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.406068 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.406072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.406075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.406111 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.408887 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.409476 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.409483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.409486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.409598 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.409965 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.409971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.409975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.410306 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.410456 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.410849 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.411050 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:22:09.417618 6613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa: Bootstrap starting.
I20260926 23:22:09.421625 6613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:09.422916 6613 log.cc:824] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa: Log is configured to *not* fsync() on all Append() calls
I20260926 23:22:09.426370 6613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa: No bootstrap required, opened a new log
I20260926 23:22:09.434208 6613 raft_consensus.cc:348] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } }
I20260926 23:22:09.434639 6613 raft_consensus.cc:729] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df86c33d09d3491bb378675636f601fa, State: Initialized, Role: FOLLOWER
I20260926 23:22:09.435137 6613 consensus_queue.cc:260] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } }
I20260926 23:22:09.436122 6782 sys_catalog.cc:455] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.436508 6782 sys_catalog.cc:458] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:09.437363 6613 sys_catalog.cc:564] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:22:09.437542 6587 raft_consensus.cc:482] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 23:22:09.437740 6587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:09.439955 6587 raft_consensus.cc:504] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } }
I20260926 23:22:09.441238 6587 leader_election.cc:288] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [CANDIDATE]: Term 1 election: Requested vote from peers 7d9b9da93637494ca63c68b7b328eda6 (127.16.62.125:46611), cca7a0c536fc49318e422a699ec8e019 (127.16.62.124:45983)
I20260926 23:22:09.442500 6737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df86c33d09d3491bb378675636f601fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cca7a0c536fc49318e422a699ec8e019"
I20260926 23:22:09.442960 6737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:09.448055 6737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df86c33d09d3491bb378675636f601fa in term 1.
I20260926 23:22:09.448979 6562 leader_election.cc:302] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca7a0c536fc49318e422a699ec8e019, df86c33d09d3491bb378675636f601fa; no voters:
I20260926 23:22:09.449506 6782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:22:09.463502 6782 raft_consensus.cc:686] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [term 1 LEADER]: Becoming Leader. State: Replica: df86c33d09d3491bb378675636f601fa, State: Running, Role: LEADER
I20260926 23:22:09.463951 6687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } attempt: 5
I20260926 23:22:09.464629 6782 consensus_queue.cc:237] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } }
Sep 26 23:22:09 dist-test-slave-vcv1 krb5kdc[6532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464928, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
W20260926 23:22:09.477491 6715 tablet.cc:2369] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:22:09.488628 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.464622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:45983 (local address 127.0.0.1:52424)
0926 23:22:09.465021 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:09.465043 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:09.465180 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:09.466141 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:09.466146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:09.466639 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:09.466868 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.466880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.468337 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.468342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.470151 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:09.470169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.471622 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.471631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:09.471956 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.472805 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:09.472836 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:09.475978 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:09.482940 (+ 6962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.482972 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.482979 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.484126 (+ 1147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.485058 (+ 932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:09.485065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:09.485097 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:09.485436 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:09.486660 (+ 1224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:09.486667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:09.486937 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 23:22:09.487149 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 23:22:09.487756 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260926 23:22:09.491223 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.465402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.16.62.124:45983)
0926 23:22:09.465579 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.465584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.465603 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:22:09.465662 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.465667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.465768 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.465870 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.467072 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.468166 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.470360 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.471222 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.471364 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.476175 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.476184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.476189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.476226 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.482629 (+ 6403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.484429 (+ 1800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.484437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.484473 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.484829 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.485697 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.485752 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.485758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.486374 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.489391 (+ 3017us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.489817 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.490307 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:22:09.497166 6687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6: Bootstrap starting.
W20260926 23:22:09.500030 6566 tablet.cc:2369] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:22:09.502290 6687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:09.503568 6687 log.cc:824] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6: Log is configured to *not* fsync() on all Append() calls
I20260926 23:22:09.513980 6788 sys_catalog.cc:455] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader df86c33d09d3491bb378675636f601fa. Latest consensus state: current_term: 1 leader_uuid: "df86c33d09d3491bb378675636f601fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.514472 6788 sys_catalog.cc:458] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:09.519412 6687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6: No bootstrap required, opened a new log
I20260926 23:22:09.517210 6545 master_runner.cc:186] Error getting master registration for 127.16.62.125:46611: OK, instance_id { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" instance_seqno: 1790464928740067 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:22:09.528795 6545 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:22:09.550680 6687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } }
I20260926 23:22:09.551215 6687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9b9da93637494ca63c68b7b328eda6, State: Initialized, Role: FOLLOWER
I20260926 23:22:09.551863 6687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } }
I20260926 23:22:09.562304 6806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.563047 6806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:09.563560 6687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:22:09.563623 6662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df86c33d09d3491bb378675636f601fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d9b9da93637494ca63c68b7b328eda6"
I20260926 23:22:09.564071 6662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:09.568697 6662 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df86c33d09d3491bb378675636f601fa in term 1.
I20260926 23:22:09.569262 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.486965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.16.62.125:46611)
0926 23:22:09.487149 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.487156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.536503 (+ 49347us) server_negotiation.cc:407] Connection header received
0926 23:22:09.536768 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.536774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.536868 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.536939 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.539150 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.540335 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.542246 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.542949 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.543050 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.552326 (+ 9276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.552332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.552336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.552374 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.556035 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.565229 (+ 9194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.565238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.565243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.565443 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.566670 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.566676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.566679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.567067 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.567718 (+ 651us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.568332 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.568602 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 23:22:09.570693 6802 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:22:09.570931 6802 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:22:09.571130 6803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:22:09.573464 6803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:22:09.582211 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.480278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.16.62.126:40221)
0926 23:22:09.480451 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.480458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.515953 (+ 35495us) server_negotiation.cc:407] Connection header received
0926 23:22:09.521515 (+ 5562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.521523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.521627 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.521717 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.532743 (+ 11026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.533913 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.548260 (+ 14347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.549051 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.549153 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.569308 (+ 20155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.569315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.569320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.569366 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.574795 (+ 5429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.577546 (+ 2751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.577554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.577558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.577769 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.578210 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.578215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.578219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.578589 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.578835 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.581578 (+ 2743us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.581823 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:22:09.590128 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:09.517404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.16.62.124:45983)
0926 23:22:09.517631 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.517639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.546998 (+ 29359us) server_negotiation.cc:407] Connection header received
0926 23:22:09.547247 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.547253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.547370 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.547441 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.561160 (+ 13719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.562312 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.575717 (+ 13405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:09.576448 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:09.576568 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:09.583135 (+ 6567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:09.583147 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:09.583152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:09.583194 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:09.586926 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.587725 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.587731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.587735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.587869 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:09.588325 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:09.588330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:09.588334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:09.588710 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:09.588954 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:09.589483 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 23:22:09.589755 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:22:09.593750 6737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [term 1 FOLLOWER]: Refusing update from remote peer df86c33d09d3491bb378675636f601fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:22:09.594763 6788 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:22:09.596750 6662 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [term 1 FOLLOWER]: Refusing update from remote peer df86c33d09d3491bb378675636f601fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:22:09.602387 6788 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:22:09.638164 6768 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df86c33d09d3491bb378675636f601fa. Latest consensus state: current_term: 1 leader_uuid: "df86c33d09d3491bb378675636f601fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.638509 6768 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:09.641961 6808 mvcc.cc:205] Tried to move back new op lower bound from 7333744351603351552 to 7333744351096578048. Current Snapshot: MvccSnapshot[applied={T|T < 7333744351603351552}]
I20260926 23:22:09.653458 6806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df86c33d09d3491bb378675636f601fa. Latest consensus state: current_term: 1 leader_uuid: "df86c33d09d3491bb378675636f601fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.653857 6806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:09.664786 6806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df86c33d09d3491bb378675636f601fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.665089 6806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:09.670518 6812 mvcc.cc:205] Tried to move back new op lower bound from 7333744351603351552 to 7333744351096578048. Current Snapshot: MvccSnapshot[applied={T|T < 7333744351603351552}]
I20260926 23:22:09.672546 6811 mvcc.cc:205] Tried to move back new op lower bound from 7333744351603351552 to 7333744351096578048. Current Snapshot: MvccSnapshot[applied={T|T < 7333744351603351552 or (T in {7333744351603351552})}]
I20260926 23:22:09.681241 6768 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df86c33d09d3491bb378675636f601fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.681777 6768 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:09.683468 6803 catalog_manager.cc:1347] Generated new cluster ID: cb1202c261d040bfba174b41629271cf
I20260926 23:22:09.683573 6803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:22:09.698153 6782 sys_catalog.cc:455] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df86c33d09d3491bb378675636f601fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.698472 6782 sys_catalog.cc:458] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:09.698840 6782 sys_catalog.cc:455] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df86c33d09d3491bb378675636f601fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86c33d09d3491bb378675636f601fa" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 40221 } } peers { permanent_uuid: "7d9b9da93637494ca63c68b7b328eda6" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 46611 } } peers { permanent_uuid: "cca7a0c536fc49318e422a699ec8e019" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 45983 } } }
I20260926 23:22:09.699115 6782 sys_catalog.cc:458] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:09.725725 6825 catalog_manager.cc:1259] Loaded cluster ID: cb1202c261d040bfba174b41629271cf
I20260926 23:22:09.725940 6825 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6: loading cluster ID for follower catalog manager: success
W20260926 23:22:09.727298 6825 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 23:22:09.727519 6825 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:22:09.789307 6803 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:22:09.791179 6803 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:22:09.831394 6803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df86c33d09d3491bb378675636f601fa: Generated new TSK 0
I20260926 23:22:09.832536 6803 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:22:09.925158 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:09.923036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.16.62.125:46611)
0926 23:22:09.923227 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.923233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.923257 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:22:09.923523 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.923530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.923617 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.923692 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.924895 (+ 1203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50410: BlockingRecv error: recv got EOF from 127.0.0.1:50410 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 23:22:09.927205 6616 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:09.922879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.16.62.126:40221)
0926 23:22:09.923123 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.923129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.925659 (+ 2530us) server_negotiation.cc:407] Connection header received
0926 23:22:09.925891 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.925896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.925969 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.926052 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.926966 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40750: BlockingRecv error: recv got EOF from 127.0.0.1:40750 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 23:22:09.931797 6765 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:09.930268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.16.62.124:45983)
0926 23:22:09.930421 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.930426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.930443 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:22:09.930562 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.930567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.930634 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.930733 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.931587 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52438: BlockingRecv error: recv got EOF from 127.0.0.1:52438 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 23:22:09.939049 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:09.936954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.16.62.125:46611)
0926 23:22:09.937138 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.937144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.937630 (+ 486us) server_negotiation.cc:407] Connection header received
0926 23:22:09.937845 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.937852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.937975 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.938087 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.938792 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50412: BlockingRecv error: recv got EOF from 127.0.0.1:50412 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 23:22:09.945751 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:09.944003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.16.62.125:46611)
0926 23:22:09.944161 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.944166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.944407 (+ 241us) server_negotiation.cc:407] Connection header received
0926 23:22:09.944593 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.944598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.944682 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.944792 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.945535 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50420: BlockingRecv error: recv got EOF from 127.0.0.1:50420 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 23:22:09.957995 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:09.951909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.16.62.125:46611)
0926 23:22:09.952081 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.952087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.956662 (+ 4575us) server_negotiation.cc:407] Connection header received
0926 23:22:09.956835 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.956841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.956939 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.957038 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.957755 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50434: BlockingRecv error: recv got EOF from 127.0.0.1:50434 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 23:22:09.966791 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:09.964727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.16.62.125:46611)
0926 23:22:09.964927 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.964932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.965429 (+ 497us) server_negotiation.cc:407] Connection header received
0926 23:22:09.965621 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.965627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.965713 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.965815 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.966513 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50446: BlockingRecv error: recv got EOF from 127.0.0.1:50446 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260926 23:22:09.980297 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:09.977923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.16.62.125:46611)
0926 23:22:09.978096 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.978102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:09.978634 (+ 532us) server_negotiation.cc:407] Connection header received
0926 23:22:09.978907 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:09.978913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:09.979005 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:09.979119 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:09.980028 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50460: BlockingRecv error: recv got EOF from 127.0.0.1:50460 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 23:22:10.002046 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:09.999735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.16.62.125:46611)
0926 23:22:09.999933 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:09.999940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:10.000516 (+ 576us) server_negotiation.cc:407] Connection header received
0926 23:22:10.000693 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:10.000699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:10.000828 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:10.000955 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:10.001757 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50464: BlockingRecv error: recv got EOF from 127.0.0.1:50464 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 23:22:10.041512 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:10.039114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.16.62.125:46611)
0926 23:22:10.039308 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:10.039315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:10.039863 (+ 548us) server_negotiation.cc:407] Connection header received
0926 23:22:10.040075 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:10.040081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:10.040217 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:10.040339 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:10.041233 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50476: BlockingRecv error: recv got EOF from 127.0.0.1:50476 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 23:22:10.111315 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:10.109170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.16.62.125:46611)
0926 23:22:10.109408 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:10.109415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:10.109770 (+ 355us) server_negotiation.cc:407] Connection header received
0926 23:22:10.109994 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:10.109999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:10.110076 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:10.110169 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:10.111055 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50478: BlockingRecv error: recv got EOF from 127.0.0.1:50478 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 23:22:10.244760 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:10.242640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.16.62.125:46611)
0926 23:22:10.242832 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:10.242838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:10.243341 (+ 503us) server_negotiation.cc:407] Connection header received
0926 23:22:10.243514 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:10.243518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:10.243597 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:10.243679 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:10.244530 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50494: BlockingRecv error: recv got EOF from 127.0.0.1:50494 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:22:10.337770 6779 catalog_manager.cc:1259] Loaded cluster ID: cb1202c261d040bfba174b41629271cf
I20260926 23:22:10.337891 6779 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019: loading cluster ID for follower catalog manager: success
I20260926 23:22:10.340406 6779 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019: acquiring CA information for follower catalog manager: success
I20260926 23:22:10.342118 6779 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cca7a0c536fc49318e422a699ec8e019: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 23:22:10.511691 6691 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:10.510030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.16.62.125:46611)
0926 23:22:10.510259 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:10.510267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:10.510457 (+ 190us) server_negotiation.cc:407] Connection header received
0926 23:22:10.510629 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:10.510633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:10.510706 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:10.510793 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:10.511483 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50504: BlockingRecv error: recv got EOF from 127.0.0.1:50504 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:22:10.730897 6825 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6: acquiring CA information for follower catalog manager: success
I20260926 23:22:10.732689 6825 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d9b9da93637494ca63c68b7b328eda6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:22:11.033318 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:11.027601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.16.62.125:46611)
0926 23:22:11.028276 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:11.028280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:11.028299 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:22:11.028364 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:11.028367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:11.028519 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:11.028627 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:11.029042 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.029916 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.031271 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.031670 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.031772 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:11.032226 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:11.032720 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:11.032834 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:11.032915 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 23:22:11.032983 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":363,"threads_started":1}
I20260926 23:22:11.043777 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:11.038730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.16.62.124:45983)
0926 23:22:11.039405 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:11.039422 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:11.039437 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:22:11.039540 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:11.039545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:11.039685 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:11.039816 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:11.040122 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.040819 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.041994 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.042361 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.042447 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:11.042843 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:11.043322 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:11.043431 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:11.043502 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 23:22:11.043561 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":481,"threads_started":1}
I20260926 23:22:11.056428 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:11.049629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.16.62.125:46611)
0926 23:22:11.049872 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:11.049880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:11.050881 (+ 1001us) server_negotiation.cc:407] Connection header received
0926 23:22:11.051063 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:11.051068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:11.051157 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:11.051250 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:11.051960 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.052912 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.054620 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.055114 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.055235 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:11.055707 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:11.055799 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:11.055894 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:11.056022 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 23:22:11.056104 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 23:22:11.057511 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:11.051489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.16.62.124:45983)
0926 23:22:11.051842 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:11.051851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:11.051873 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:22:11.051942 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:11.051949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:11.052065 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:11.052186 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:11.052679 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.053547 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.055477 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.056078 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.056235 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:11.056814 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:11.056938 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:11.057033 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:11.057170 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 23:22:11.057238 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260926 23:22:11.058926 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:11.049593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.16.62.126:40221)
0926 23:22:11.052657 (+ 3064us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:11.052662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:11.052679 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:22:11.052793 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:11.052798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:11.052981 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:11.053159 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:11.053746 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.054833 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.056897 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:11.057460 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:11.057587 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:11.058073 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:11.058354 (+ 281us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:11.058468 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:11.058571 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 23:22:11.058660 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2942,"thread_start_us":2845,"threads_started":1}
I20260926 23:22:11.063441 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6545
I20260926 23:22:11.064493 6608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:22:11.173318 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 6545
I20260926 23:22:11.193835 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6620
I20260926 23:22:11.194921 6682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:22:11.299984 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 6620
I20260926 23:22:11.319797 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6694
I20260926 23:22:11.320914 6757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:22:11.426923 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 6694
2026-09-26T23:22:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4941 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:22:11 dist-test-slave-vcv1 krb5kdc[6911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:22:11 dist-test-slave-vcv1 krb5kdc[6911](info): set up 2 sockets
Sep 26 23:22:11 dist-test-slave-vcv1 krb5kdc[6911](info): commencing operation
krb5kdc: starting...
W20260926 23:22:12.549588 16633 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/test-user.keytab.
Sep 26 23:22:12 dist-test-slave-vcv1 krb5kdc[6911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:22:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.62.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
Entry for principal kudu/127.16.62.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab.
I20260926 23:22:12.810575 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:39949
--webserver_interface=127.16.62.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:39949,127.16.62.125:41271,127.16.62.124:39623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:22:13.132638 6924 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:22:13.133006 6924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:13.133131 6924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:13.133268 6924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:13.140360 6924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:13.140488 6924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:13.140594 6924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:13.140687 6924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:13.140777 6924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:13.150399 6924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:39949,127.16.62.125:41271,127.16.62.124:39623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.126.keytab
--principal=kudu/127.16.62.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.126:39949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.62.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:13.151731 6924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:13.153051 6924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:22:13.164791 6929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:13.164835 6932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:13.164865 6930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:13.165928 6924 server_base.cc:1034] running on GCE node
I20260926 23:22:13.166625 6924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:13.167830 6924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:13.169080 6924 hybrid_clock.cc:648] HybridClock initialized: now 1790464933169027 us; error 44 us; skew 500 ppm
I20260926 23:22:13.169433 6924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:13 dist-test-slave-vcv1 krb5kdc[6911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464933, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:13.172519 6924 init.cc:377] Logged in from keytab as kudu/127.16.62.126@KRBTEST.COM (short username kudu)
I20260926 23:22:13.173898 6924 webserver.cc:460] Webserver started at http://127.16.62.126:38249/ using document root <none> and password file <none>
I20260926 23:22:13.174511 6924 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:13.174602 6924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:13.174847 6924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:13.177718 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/data/instance:
uuid: "98ae50c4523641f6ad80a1555fe62f31"
format_stamp: "Formatted at 2026-09-26 23:22:13 on dist-test-slave-vcv1"
server_key: "52bb739942dfecfc13698f12e5082553"
server_key_iv: "d6652332c91431c3d7b4b50d1de8cced"
server_key_version: "encryptionkey@0"
I20260926 23:22:13.178428 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/wal/instance:
uuid: "98ae50c4523641f6ad80a1555fe62f31"
format_stamp: "Formatted at 2026-09-26 23:22:13 on dist-test-slave-vcv1"
server_key: "52bb739942dfecfc13698f12e5082553"
server_key_iv: "d6652332c91431c3d7b4b50d1de8cced"
server_key_version: "encryptionkey@0"
I20260926 23:22:13.183255 6924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:22:13.186446 6939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:13.187691 6924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:22:13.187858 6924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/wal
uuid: "98ae50c4523641f6ad80a1555fe62f31"
format_stamp: "Formatted at 2026-09-26 23:22:13 on dist-test-slave-vcv1"
server_key: "52bb739942dfecfc13698f12e5082553"
server_key_iv: "d6652332c91431c3d7b4b50d1de8cced"
server_key_version: "encryptionkey@0"
I20260926 23:22:13.188062 6924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:13.231905 6924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:13.235921 6924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:13.236229 6924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:13.254115 6924 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.126:39949
I20260926 23:22:13.254129 6990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.126:39949 every 8 connection(s)
I20260926 23:22:13.255370 6924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/data/info.pb
I20260926 23:22:13.259022 6991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:13.263525 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 6924
I20260926 23:22:13.263720 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-0/wal/instance
I20260926 23:22:13.264189 16633 external_mini_cluster.cc:1428] Setting key 789159b368f5c6d63943a538cf220f79
I20260926 23:22:13.265391 6991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:13 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464933, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:22:13.289587 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.268474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39949 (local address 127.0.0.1:37604)
0926 23:22:13.269558 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.269595 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:13.271279 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:13.272221 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:13.272231 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:13.273028 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:13.273536 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:13.273562 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.275139 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.275148 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:13.277057 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:13.277074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.277981 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.277991 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:13.278215 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:13.278896 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:13.278928 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:13.281498 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:13.285803 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:13.285812 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:13.285820 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:13.286423 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:13.286813 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:13.286820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:13.286826 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:13.287053 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:13.287752 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:13.287759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:13.287975 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 23:22:13.288234 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 23:22:13.288593 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":672,"threads_started":1}
I20260926 23:22:13.289626 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.268469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.16.62.126:39949)
0926 23:22:13.269280 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:13.269299 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:13.269709 (+ 410us) server_negotiation.cc:407] Connection header received
0926 23:22:13.271632 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:13.271638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:13.271907 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:13.272093 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:13.273765 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.274952 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.277213 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.277735 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.277864 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:13.281759 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:13.281772 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:13.281790 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:13.281834 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:13.285672 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:13.286558 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:13.286564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:13.286566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:13.286708 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:13.287168 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:13.287172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:13.287180 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:13.287621 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:13.287822 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:13.288422 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 23:22:13.288768 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":569,"thread_start_us":359,"threads_started":1}
WARNING: no policy specified for kudu/127.16.62.125@KRBTEST.COM; defaulting to no policy
I20260926 23:22:13.292558 6991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.125@KRBTEST.COM" created.
I20260926 23:22:13.293713 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.293276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50308)
0926 23:22:13.293448 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.293573 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 23:22:13.294040 6940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
I20260926 23:22:13.295483 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.295153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50314)
0926 23:22:13.295279 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.295354 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:22:13.295779 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:41271: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab.
Entry for principal kudu/127.16.62.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab.
I20260926 23:22:13.323540 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.125:41271
--webserver_interface=127.16.62.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab
--principal=kudu/127.16.62.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:39949,127.16.62.125:41271,127.16.62.124:39623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:22:13.345067 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } attempt: 1
I20260926 23:22:13.346372 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.345741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50328)
0926 23:22:13.345999 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.346148 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 23:22:13.347421 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.347027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50340)
0926 23:22:13.347185 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.347270 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:22:13.347651 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:41271: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
I20260926 23:22:13.415901 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } attempt: 2
I20260926 23:22:13.417255 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.416587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50348)
0926 23:22:13.416912 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.417048 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 23:22:13.418421 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.417917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50356)
0926 23:22:13.418110 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.418232 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 23:22:13.418664 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:41271: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
I20260926 23:22:13.510036 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } attempt: 3
I20260926 23:22:13.511372 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.510770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50362)
0926 23:22:13.511043 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.511178 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 23:22:13.512619 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.512098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50364)
0926 23:22:13.512319 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.512417 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260926 23:22:13.512842 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:41271: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
W20260926 23:22:13.643918 6998 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:22:13.644248 6998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:13.644348 6998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:13.644520 6998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:13.651520 6998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:13.651597 6998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:13.651687 6998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:13.651782 6998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:13.651892 6998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:13.656111 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } attempt: 4
I20260926 23:22:13.657362 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.656763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50376)
0926 23:22:13.657081 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.657193 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 23:22:13.658368 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.657988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50384)
0926 23:22:13.658134 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.658228 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:22:13.658541 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.125:41271: Network error: Client connection negotiation failed: client connection to 127.16.62.125:41271: connect: Connection refused (error 111)
I20260926 23:22:13.662309 6998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:39949,127.16.62.125:41271,127.16.62.124:39623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.125.keytab
--principal=kudu/127.16.62.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.125:41271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.62.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:13.663448 6998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:13.664765 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:22:13.676776 7004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:13.676960 7003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:13.677086 7006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:13.681555 6998 server_base.cc:1034] running on GCE node
I20260926 23:22:13.682368 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:13.683733 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:13.684973 6998 hybrid_clock.cc:648] HybridClock initialized: now 1790464933684918 us; error 48 us; skew 500 ppm
I20260926 23:22:13.685299 6998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:13 dist-test-slave-vcv1 krb5kdc[6911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464933, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:13.688333 6998 init.cc:377] Logged in from keytab as kudu/127.16.62.125@KRBTEST.COM (short username kudu)
I20260926 23:22:13.689787 6998 webserver.cc:460] Webserver started at http://127.16.62.125:42651/ using document root <none> and password file <none>
I20260926 23:22:13.690483 6998 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:13.690613 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:13.690917 6998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:13.693830 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/data/instance:
uuid: "bfc0f0b5980546bca895ee383bb5d78f"
format_stamp: "Formatted at 2026-09-26 23:22:13 on dist-test-slave-vcv1"
server_key: "a50e97bddf4f8048e179319434976110"
server_key_iv: "793a5d3085e1e12124d43344e155db51"
server_key_version: "encryptionkey@0"
I20260926 23:22:13.694617 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/wal/instance:
uuid: "bfc0f0b5980546bca895ee383bb5d78f"
format_stamp: "Formatted at 2026-09-26 23:22:13 on dist-test-slave-vcv1"
server_key: "a50e97bddf4f8048e179319434976110"
server_key_iv: "793a5d3085e1e12124d43344e155db51"
server_key_version: "encryptionkey@0"
I20260926 23:22:13.699731 6998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:22:13.703073 7013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:13.704358 6998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:22:13.704530 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/wal
uuid: "bfc0f0b5980546bca895ee383bb5d78f"
format_stamp: "Formatted at 2026-09-26 23:22:13 on dist-test-slave-vcv1"
server_key: "a50e97bddf4f8048e179319434976110"
server_key_iv: "793a5d3085e1e12124d43344e155db51"
server_key_version: "encryptionkey@0"
I20260926 23:22:13.704735 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:13.724632 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:13.728423 6998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:13.728674 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:13.753233 6998 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.125:41271
I20260926 23:22:13.753237 7064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.125:41271 every 8 connection(s)
I20260926 23:22:13.754916 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/data/info.pb
I20260926 23:22:13.757136 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 6998
I20260926 23:22:13.757316 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-1/wal/instance
I20260926 23:22:13.757899 16633 external_mini_cluster.cc:1428] Setting key 8f24bd97f565aa62cb531bbe1ebd4b3a
I20260926 23:22:13.759562 7065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:13.769148 7065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:13 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464933, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.62.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.62.124@KRBTEST.COM" created.
I20260926 23:22:13.792532 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.772412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.16.62.126:39949)
0926 23:22:13.772671 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:13.772679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:13.773618 (+ 939us) server_negotiation.cc:407] Connection header received
0926 23:22:13.776525 (+ 2907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:13.776532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:13.776639 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:13.776726 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:13.779061 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.780221 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.782259 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.783002 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.783178 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:13.786581 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:13.786590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:13.786598 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:13.786643 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:13.789435 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:13.790227 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:13.790232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:13.790234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:13.790351 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:13.790803 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:13.790806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:13.790808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:13.791113 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:13.791225 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:13.792041 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 23:22:13.792299 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 23:22:13.793889 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.772633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39949 (local address 127.0.0.1:37612)
0926 23:22:13.773441 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.773492 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:13.776170 (+ 2678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:13.777295 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:13.777306 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:13.778096 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:13.778868 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:13.778895 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.780388 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.780397 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:13.782093 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:13.782116 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.783238 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.783243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:13.783440 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:13.784219 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:13.784241 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:13.786347 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:13.789562 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:13.789568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:13.789579 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:13.790112 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:13.790464 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:13.790471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:13.790478 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:13.790698 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:13.791214 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:13.791223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:13.791661 (+ 438us) client_negotiation.cc:769] Sending connection context
0926 23:22:13.791953 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 23:22:13.792515 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":336,"threads_started":1}
I20260926 23:22:13.795522 7065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:13 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464933, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:22:13.809897 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.796356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50394)
0926 23:22:13.796523 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.796542 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:13.796635 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:13.797533 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:13.797538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:13.797952 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:13.798066 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:13.798074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.799408 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.799416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:13.800327 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:13.800338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.801128 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.801134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:13.801302 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:13.801856 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:13.801872 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:13.803977 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:13.807404 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:13.807409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:13.807413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:13.807822 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:13.808230 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:13.808236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:13.808240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:13.808376 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:13.808955 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:13.808962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:13.809110 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 23:22:13.809258 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 23:22:13.809522 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 23:22:13.810032 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.796593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.16.62.125:41271)
0926 23:22:13.797173 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:13.797179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:13.797201 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:22:13.797239 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:13.797244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:13.797398 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:13.797546 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:13.798241 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.799278 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.800457 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.800963 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.801106 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:13.804224 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:13.804234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:13.804245 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:13.804271 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:13.807270 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:13.807955 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:13.807963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:13.807967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:13.808081 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:13.808484 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:13.808492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:13.808496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:13.808839 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:13.809049 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:13.809389 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 23:22:13.809646 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":438,"thread_start_us":319,"threads_started":1}
I20260926 23:22:13.812412 7065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:22:13.813434 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.813022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49616)
0926 23:22:13.813197 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.813297 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:22:13.813671 7014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
I20260926 23:22:13.815160 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.814750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49618)
0926 23:22:13.814900 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.815065 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 23:22:13.815366 7065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:39623: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.62.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab.
Entry for principal kudu/127.16.62.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab.
I20260926 23:22:13.822928 16633 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
/tmp/dist-test-taskZEqmPv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.124:39623
--webserver_interface=127.16.62.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab
--principal=kudu/127.16.62.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.62.84:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.62.126:39949,127.16.62.125:41271,127.16.62.124:39623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:22:13.864692 7065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } attempt: 1
I20260926 23:22:13.865909 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.865466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49628)
0926 23:22:13.865653 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.865763 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 23:22:13.866911 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.866578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49634)
0926 23:22:13.866742 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.866811 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:22:13.867151 7065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:39623: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
I20260926 23:22:13.935391 7065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } attempt: 2
I20260926 23:22:13.936686 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.936095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49650)
0926 23:22:13.936381 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.936493 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 23:22:13.937757 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.937364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49654)
0926 23:22:13.937513 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.937597 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:22:13.938022 7065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:39623: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
I20260926 23:22:13.957772 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } attempt: 5
Sep 26 23:22:13 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464933, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:22:13.972687 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.958507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50398)
0926 23:22:13.958699 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.958724 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:13.958834 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:13.959217 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:13.959220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:13.959584 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:13.959757 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:13.959764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.961260 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.961264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:13.962121 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:13.962132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.963122 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.963127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:13.963315 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:13.963963 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:13.963989 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:13.966333 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:13.970082 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:13.970086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:13.970089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:13.970495 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:13.970923 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:13.970927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:13.970929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:13.971065 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:13.971807 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:13.971811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:13.972020 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 23:22:13.972203 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 23:22:13.972426 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 23:22:13.973060 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.958595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.16.62.125:41271)
0926 23:22:13.958803 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:13.958810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:13.958834 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:22:13.958973 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:13.958979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:13.959062 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:13.959187 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:13.959931 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.961109 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.962268 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:13.962947 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:13.963100 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:13.966534 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:13.966540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:13.966544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:13.966582 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:13.969900 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:13.970627 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:13.970632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:13.970636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:13.970795 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:13.971250 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:13.971255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:13.971258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:13.971667 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:13.971872 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:13.972353 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 23:22:13.972636 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 23:22:13.973989 6991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:22:13.975059 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.974631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49662)
0926 23:22:13.974850 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.974920 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 23:22:13.976056 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:13.975706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49668)
0926 23:22:13.975870 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:13.975937 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:22:13.976320 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:39623: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
I20260926 23:22:14.027540 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } attempt: 1
I20260926 23:22:14.028846 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.028360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49670)
0926 23:22:14.028585 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.028679 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 23:22:14.029203 7065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } attempt: 3
I20260926 23:22:14.030208 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.029750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49674)
0926 23:22:14.029985 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.030073 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 23:22:14.030224 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.029724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49676)
0926 23:22:14.029940 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.030054 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 23:22:14.030503 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:39623: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
I20260926 23:22:14.031462 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.031038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49694)
0926 23:22:14.031269 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.031341 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260926 23:22:14.031764 7065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:39623: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
I20260926 23:22:14.098749 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } attempt: 2
I20260926 23:22:14.099977 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.099374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49702)
0926 23:22:14.099650 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.099767 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 23:22:14.101159 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.100758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49718)
0926 23:22:14.100936 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.101029 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 23:22:14.101395 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:39623: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
W20260926 23:22:14.152238 7072 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:22:14.152524 7072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:22:14.152587 7072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:22:14.152666 7072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:22:14.159539 7072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:22:14.159623 7072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:22:14.159679 7072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:22:14.159721 7072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:22:14.159782 7072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:22:14.169199 7072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.62.84:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.62.126:39949,127.16.62.125:41271,127.16.62.124:39623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/krb5kdc/kudu_127.16.62.124.keytab
--principal=kudu/127.16.62.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.62.124:39623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.62.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:22:14.170341 7072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:22:14.171543 7072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:22:14.175038 7065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } attempt: 4
I20260926 23:22:14.176438 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.175802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49720)
0926 23:22:14.176059 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.176231 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 23:22:14.177733 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.177232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49724)
0926 23:22:14.177486 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.177593 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260926 23:22:14.178066 7065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:39623: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
W20260926 23:22:14.183367 7077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:14.183436 7078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:22:14.183408 7080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:22:14.184243 7072 server_base.cc:1034] running on GCE node
I20260926 23:22:14.184990 7072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:22:14.186244 7072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:22:14.187539 7072 hybrid_clock.cc:648] HybridClock initialized: now 1790464934187463 us; error 51 us; skew 500 ppm
I20260926 23:22:14.187898 7072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:22:14 dist-test-slave-vcv1 krb5kdc[6911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:22:14.191020 7072 init.cc:377] Logged in from keytab as kudu/127.16.62.124@KRBTEST.COM (short username kudu)
I20260926 23:22:14.192605 7072 webserver.cc:460] Webserver started at http://127.16.62.124:40677/ using document root <none> and password file <none>
I20260926 23:22:14.193238 7072 fs_manager.cc:359] Metadata directory not provided
I20260926 23:22:14.193344 7072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:22:14.193675 7072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:22:14.196573 7072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/data/instance:
uuid: "691c1195519344729ca2470f8e2f0f0c"
format_stamp: "Formatted at 2026-09-26 23:22:14 on dist-test-slave-vcv1"
server_key: "2b7a808825a7bbb789852b2e28243547"
server_key_iv: "c299b0342948850a60551ed6ed077638"
server_key_version: "encryptionkey@0"
I20260926 23:22:14.197340 7072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/wal/instance:
uuid: "691c1195519344729ca2470f8e2f0f0c"
format_stamp: "Formatted at 2026-09-26 23:22:14 on dist-test-slave-vcv1"
server_key: "2b7a808825a7bbb789852b2e28243547"
server_key_iv: "c299b0342948850a60551ed6ed077638"
server_key_version: "encryptionkey@0"
I20260926 23:22:14.202225 7072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:22:14.205438 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:22:14.206674 7072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:22:14.206851 7072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/data,/tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/wal
uuid: "691c1195519344729ca2470f8e2f0f0c"
format_stamp: "Formatted at 2026-09-26 23:22:14 on dist-test-slave-vcv1"
server_key: "2b7a808825a7bbb789852b2e28243547"
server_key_iv: "c299b0342948850a60551ed6ed077638"
server_key_version: "encryptionkey@0"
I20260926 23:22:14.207059 7072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:22:14.207621 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } attempt: 3
I20260926 23:22:14.208979 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.208389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49740)
0926 23:22:14.208664 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.208784 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 23:22:14.209929 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.209580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49754)
0926 23:22:14.209759 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.209835 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 23:22:14.210125 6991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.62.124:39623: Network error: Client connection negotiation failed: client connection to 127.16.62.124:39623: connect: Connection refused (error 111)
I20260926 23:22:14.249464 7072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:22:14.253497 7072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:22:14.253806 7072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:22:14.272321 7072 rpc_server.cc:307] RPC server started. Bound to: 127.16.62.124:39623
I20260926 23:22:14.272297 7139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.62.124:39623 every 8 connection(s)
I20260926 23:22:14.273577 7072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/data/info.pb
I20260926 23:22:14.276830 16633 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu as pid 7072
I20260926 23:22:14.277045 16633 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZEqmPv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464575313384-16633-0/minicluster-data/master-2/wal/instance
I20260926 23:22:14.277216 7140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:22:14.277566 16633 external_mini_cluster.cc:1428] Setting key 0150aaa20f8d919da3af0104020e1f6d
I20260926 23:22:14.286998 7140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:14 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
Sep 26 23:22:14 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
I20260926 23:22:14.301342 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.279222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.16.62.124:39623)
0926 23:22:14.280022 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.280036 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.280092 (+ 56us) server_negotiation.cc:407] Connection header received
0926 23:22:14.282035 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.282062 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.282562 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.282989 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.284310 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.286254 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.287551 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.288550 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.288731 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.293842 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.293850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.293859 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.293899 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.297083 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.297801 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.297807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.297813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.297981 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.298395 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.298398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.298400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.298898 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.299241 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.299540 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.299933 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":316,"threads_started":1}
I20260926 23:22:14.305233 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.289569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.126:39949 (local address 127.0.0.1:37626)
0926 23:22:14.290344 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.290394 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:14.290567 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:14.290912 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:14.290919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:14.291756 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:14.292102 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.292111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.293634 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.293638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.294888 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.294910 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.295706 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.295714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.295893 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.296534 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:14.296556 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:14.298795 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:14.302574 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.302582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.302587 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.303024 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.303449 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.303455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.303472 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.303602 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.304121 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:14.304129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:14.304347 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 23:22:14.304546 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 23:22:14.304855 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":380,"threads_started":1}
I20260926 23:22:14.305474 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.289593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.16.62.126:39949)
0926 23:22:14.289835 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.289845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.290512 (+ 667us) server_negotiation.cc:407] Connection header received
0926 23:22:14.290691 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.290695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.290755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.290831 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.292320 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.293451 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.295069 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.295495 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.295632 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.298989 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.298997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.299005 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.299050 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.302419 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.303166 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.303174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.303180 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.303321 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.303713 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.303719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.303722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.303993 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.304147 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.304836 (+ 689us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.305065 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:22:14.307004 7140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:14 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.126@KRBTEST.COM
Sep 26 23:22:14 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:22:14.321588 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.304324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.16.62.126:39949)
0926 23:22:14.304892 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.304896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.304914 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:22:14.304959 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.304962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.305013 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.305129 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.306632 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.307538 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.309505 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.310123 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.310246 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.314992 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.314998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.315001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.315070 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.318635 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.319240 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.319245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.319249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.319415 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.319830 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.319836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.319839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.320235 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.320540 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.320885 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.321132 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":292,"threads_started":1}
I20260926 23:22:14.322075 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.307935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.16.62.125:41271)
0926 23:22:14.308130 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.308184 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.308203 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:22:14.308263 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.308267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.308325 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.308399 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.309342 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.310387 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.312256 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.312657 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.312753 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.315778 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.315787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.315794 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.315847 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.319277 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.319873 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.319881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.319884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.320021 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.320496 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.320502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.320505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.320909 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.321069 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.321507 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.321710 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:22:14.322381 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.307742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.125:41271 (local address 127.0.0.1:50414)
0926 23:22:14.307933 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.307956 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:14.308059 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:14.308487 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:14.308493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:14.308982 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:14.309176 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.309189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.310533 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.310539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.312052 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.312069 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.312813 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.312820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.313018 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.313530 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:14.313546 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:14.315546 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:14.319400 (+ 3854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.319404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.319406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.319744 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.320163 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.320170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.320174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.320388 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.321041 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:14.321047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:14.321209 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 23:22:14.321379 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 23:22:14.321688 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 23:22:14.323329 7140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:22:14 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.124@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
Sep 26 23:22:14 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.62.125@KRBTEST.COM
I20260926 23:22:14.338358 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.324233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.16.62.124:39623)
0926 23:22:14.324405 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.324409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.324425 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:22:14.324491 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.324494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.324551 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.324630 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.325512 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.326593 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.328315 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.328972 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.329109 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.332783 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.332788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.332791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.332817 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.335580 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.336413 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.336416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.336418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.336527 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.336946 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.336949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.336951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.337208 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.337306 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.337747 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.337981 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:22:14.338622 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.324001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49768)
0926 23:22:14.324280 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.324300 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:14.324395 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:14.324680 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:14.324689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:14.325150 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:14.325319 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.325332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.326736 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.326741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.328159 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.328174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.329161 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.329167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.329387 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.330052 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:14.330075 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:14.332609 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:14.335726 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.335735 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.335740 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.336297 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.336650 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.336656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.336661 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.336835 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.337304 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:14.337312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:14.337483 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 23:22:14.337664 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 23:22:14.338115 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":159}
I20260926 23:22:14.340240 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.323418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.16.62.125:41271)
0926 23:22:14.323569 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.323574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.323645 (+ 71us) server_negotiation.cc:407] Connection header received
0926 23:22:14.323815 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.323822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.323898 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.324010 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.325550 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.326615 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.328324 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.329001 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.329114 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.333938 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.333943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.333946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.333977 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.337092 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.337910 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.337915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.337918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.338104 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.338584 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.338593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.338597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.338935 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.339253 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.339585 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.339815 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:22:14.344269 7140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c: Bootstrap starting.
I20260926 23:22:14.345034 16633 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:22:14.347540 7140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:14.348619 7140 log.cc:824] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:22:14.347229 6966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "98ae50c4523641f6ad80a1555fe62f31"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37636
I20260926 23:22:14.351202 7140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c: No bootstrap required, opened a new log
I20260926 23:22:14.358120 7140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } }
I20260926 23:22:14.358448 7140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691c1195519344729ca2470f8e2f0f0c, State: Initialized, Role: FOLLOWER
I20260926 23:22:14.358862 7140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } }
I20260926 23:22:14.359766 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.360118 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:14.360652 7140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:22:14.369699 7158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:22:14.369825 7158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:22:14.367631 7072 master_runner.cc:186] Error getting master registration for 127.16.62.126:39949: OK, instance_id { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" instance_seqno: 1790464933249088 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:22:14.375453 7072 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:22:14.387382 6991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } attempt: 4
Sep 26 23:22:14 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464933, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.126@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
I20260926 23:22:14.403520 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.388094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49774)
0926 23:22:14.388322 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.388346 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:14.388451 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:14.388828 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:14.388839 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:14.389430 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:14.389644 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.389655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.391154 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.391159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.392659 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.392675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.393617 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.393625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.393833 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.394521 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:14.394540 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:14.396655 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:14.400752 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.400759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.400767 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.401274 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.401700 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.401704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.401708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.401871 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.402519 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:14.402524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:14.402700 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 23:22:14.402835 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 23:22:14.403111 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 23:22:14.403581 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.388233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.16.62.124:39623)
0926 23:22:14.388460 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.388466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.388514 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:22:14.388598 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.388604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.388682 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.388779 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.389812 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.391008 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.392804 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.393474 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.393617 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.396835 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.396842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.396848 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.396885 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.400570 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.401418 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.401424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.401428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.401585 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.401997 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.402004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.402012 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.402390 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.402558 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.402987 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.403204 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 23:22:14.408555 6991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31: Bootstrap starting.
I20260926 23:22:14.412600 6991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:14.413818 6991 log.cc:824] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31: Log is configured to *not* fsync() on all Append() calls
I20260926 23:22:14.416597 6991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31: No bootstrap required, opened a new log
I20260926 23:22:14.422027 6991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } }
I20260926 23:22:14.422422 6991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ae50c4523641f6ad80a1555fe62f31, State: Initialized, Role: FOLLOWER
I20260926 23:22:14.422832 6991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } }
I20260926 23:22:14.423681 7161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.424110 7161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:14.424851 6991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:22:14.424960 6966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 23:22:14.425168 6966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:14.428634 6966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } }
I20260926 23:22:14.430028 6966 leader_election.cc:288] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [CANDIDATE]: Term 1 election: Requested vote from peers bfc0f0b5980546bca895ee383bb5d78f (127.16.62.125:41271), 691c1195519344729ca2470f8e2f0f0c (127.16.62.124:39623)
I20260926 23:22:14.431356 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98ae50c4523641f6ad80a1555fe62f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "691c1195519344729ca2470f8e2f0f0c"
I20260926 23:22:14.431854 7115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:14.436391 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ae50c4523641f6ad80a1555fe62f31 in term 1.
I20260926 23:22:14.437054 6940 leader_election.cc:302] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 691c1195519344729ca2470f8e2f0f0c, 98ae50c4523641f6ad80a1555fe62f31; no voters:
I20260926 23:22:14.437547 7161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:22:14.438184 7161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [term 1 LEADER]: Becoming Leader. State: Replica: 98ae50c4523641f6ad80a1555fe62f31, State: Running, Role: LEADER
I20260926 23:22:14.438786 7161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } }
I20260926 23:22:14.442672 7169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ae50c4523641f6ad80a1555fe62f31. Latest consensus state: current_term: 1 leader_uuid: "98ae50c4523641f6ad80a1555fe62f31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.443141 7169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:14.449210 7177 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:22:14.462139 7172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:22:14.453871 6924 master_runner.cc:186] Error getting master registration for 127.16.62.125:41271: OK, instance_id { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" instance_seqno: 1790464933743178 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:22:14.464692 6924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:22:14.471276 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.454744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.16.62.125:41271)
0926 23:22:14.454915 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.454921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.455040 (+ 119us) server_negotiation.cc:407] Connection header received
0926 23:22:14.455283 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.455289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.455360 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.455443 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.457658 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.458776 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.460748 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.461435 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.461536 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.464964 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.464969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.464972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.465001 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.468368 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.469222 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.469243 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.469247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.469400 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.469806 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.469810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.469814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.470109 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.470353 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.470668 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.470893 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:22:14.477634 7065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } attempt: 5
I20260926 23:22:14.485803 7172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 23:22:14.489042 6945 tablet.cc:2369] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:22:14.489156 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.460857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.16.62.126:39949)
0926 23:22:14.461779 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.461784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.471150 (+ 9366us) server_negotiation.cc:407] Connection header received
0926 23:22:14.471403 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.471408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.471514 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.471582 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.473681 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.474777 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.476652 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.477373 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.477476 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.481846 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.481852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.481857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.481897 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.485710 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.486690 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.486697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.486701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.486860 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.487324 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.487331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.487336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.487687 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.487935 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.488510 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.488750 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797}
I20260926 23:22:14.490342 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.455636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.16.62.124:39623)
0926 23:22:14.455823 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.455829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.460699 (+ 4870us) server_negotiation.cc:407] Connection header received
0926 23:22:14.460976 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.460982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.461083 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.461163 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.473976 (+ 12813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.475108 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.476911 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.477660 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.477783 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.481526 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.481534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.481542 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.481598 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.485757 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.486982 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.487020 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.487024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.487285 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.487956 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.487961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.487994 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.488592 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.488941 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.489744 (+ 803us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.489961 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 23:22:14 dist-test-slave-vcv1 krb5kdc[6911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464933, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.62.125@KRBTEST.COM for kudu/127.16.62.124@KRBTEST.COM
I20260926 23:22:14.504707 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.478148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.62.124:39623 (local address 127.0.0.1:49798)
0926 23:22:14.478405 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 23:22:14.478424 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:22:14.478521 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:22:14.490861 (+ 12340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:22:14.490869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:22:14.491397 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:22:14.491613 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.491626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.493031 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.493037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.493890 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:22:14.493901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.494926 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.494933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:22:14.495133 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.495690 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:22:14.495707 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:22:14.498138 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:22:14.502002 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.502007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.502021 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.502496 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.502910 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:22:14.502927 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:22:14.502930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:22:14.503077 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:22:14.503670 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:22:14.503675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:22:14.503826 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 23:22:14.504002 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 23:22:14.504311 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260926 23:22:14.505366 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:14.478222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.16.62.124:39623)
0926 23:22:14.490555 (+ 12333us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.490561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.490578 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:22:14.490623 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.490628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.490697 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.490812 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.491795 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.492895 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.494058 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:14.494764 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:14.494885 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:14.498337 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:22:14.498343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:22:14.498347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:22:14.498385 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:22:14.501845 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.502647 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.502651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.502655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.502790 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:22:14.503204 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:22:14.503208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:22:14.503212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:22:14.503543 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:22:14.504274 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:14.504531 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 23:22:14.504929 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12235,"threads_started":1}
W20260926 23:22:14.505975 7093 tablet.cc:2369] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:22:14.514914 7065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f: Bootstrap starting.
I20260926 23:22:14.519812 7065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f: Neither blocks nor log segments found. Creating new log.
I20260926 23:22:14.521617 7065 log.cc:824] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:22:14.529059 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [term 1 FOLLOWER]: Refusing update from remote peer 98ae50c4523641f6ad80a1555fe62f31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:22:14.530086 7169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:22:14.533958 7065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f: No bootstrap required, opened a new log
I20260926 23:22:14.551554 7065 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } }
I20260926 23:22:14.552258 7065 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc0f0b5980546bca895ee383bb5d78f, State: Initialized, Role: FOLLOWER
I20260926 23:22:14.553046 7065 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } }
I20260926 23:22:14.563339 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ae50c4523641f6ad80a1555fe62f31. Latest consensus state: current_term: 1 leader_uuid: "98ae50c4523641f6ad80a1555fe62f31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.563638 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:14.564724 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98ae50c4523641f6ad80a1555fe62f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bfc0f0b5980546bca895ee383bb5d78f"
I20260926 23:22:14.565085 7039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:22:14.568673 7039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [term 1 FOLLOWER]: Refusing update from remote peer 98ae50c4523641f6ad80a1555fe62f31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:22:14.568866 7185 mvcc.cc:205] Tried to move back new op lower bound from 7333744371812364288 to 7333744371466956800. Current Snapshot: MvccSnapshot[applied={T|T < 7333744371812364288}]
I20260926 23:22:14.569662 7065 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:22:14.570798 7187 mvcc.cc:205] Tried to move back new op lower bound from 7333744371812364288 to 7333744371466956800. Current Snapshot: MvccSnapshot[applied={T|T < 7333744371812364288}]
I20260926 23:22:14.571193 7189 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.571391 7161 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:22:14.571623 7189 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:14.575439 7040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ae50c4523641f6ad80a1555fe62f31 in term 1.
I20260926 23:22:14.576961 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98ae50c4523641f6ad80a1555fe62f31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.577220 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:14.589433 7161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98ae50c4523641f6ad80a1555fe62f31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.589745 7161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:14.592540 7172 catalog_manager.cc:1347] Generated new cluster ID: 5339ffec62824523a0332f7e916fc2a8
I20260926 23:22:14.592664 7172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:22:14.606937 7172 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:22:14.613032 7172 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:22:14.624464 7189 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ae50c4523641f6ad80a1555fe62f31. Latest consensus state: current_term: 1 leader_uuid: "98ae50c4523641f6ad80a1555fe62f31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.624814 7189 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:14.624372 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98ae50c4523641f6ad80a1555fe62f31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.625013 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:22:14.626120 7169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98ae50c4523641f6ad80a1555fe62f31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ae50c4523641f6ad80a1555fe62f31" member_type: VOTER last_known_addr { host: "127.16.62.126" port: 39949 } } peers { permanent_uuid: "bfc0f0b5980546bca895ee383bb5d78f" member_type: VOTER last_known_addr { host: "127.16.62.125" port: 41271 } } peers { permanent_uuid: "691c1195519344729ca2470f8e2f0f0c" member_type: VOTER last_known_addr { host: "127.16.62.124" port: 39623 } } }
I20260926 23:22:14.626416 7169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31 [sys.catalog]: This master's current role is: LEADER
I20260926 23:22:14.654407 7172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ae50c4523641f6ad80a1555fe62f31: Generated new TSK 0
I20260926 23:22:14.655769 7172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:22:14.657087 7208 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 23:22:14.743196 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.738510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.16.62.125:41271)
0926 23:22:14.738681 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.738686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.740656 (+ 1970us) server_negotiation.cc:407] Connection header received
0926 23:22:14.741504 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.741511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.741597 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.741679 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.742945 (+ 1266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50438: BlockingRecv error: recv got EOF from 127.0.0.1:50438 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 23:22:14.746773 7145 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.738142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.16.62.126:39949)
0926 23:22:14.738341 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.738347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.744798 (+ 6451us) server_negotiation.cc:407] Connection header received
0926 23:22:14.745443 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.745449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.745529 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.745597 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.746509 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37648: BlockingRecv error: recv got EOF from 127.0.0.1:37648 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 23:22:14.750304 7184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.741193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.16.62.124:39623)
0926 23:22:14.741396 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.741405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.747996 (+ 6591us) server_negotiation.cc:407] Connection header received
0926 23:22:14.748618 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.748624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.748708 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.748785 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.749939 (+ 1154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49808: BlockingRecv error: recv got EOF from 127.0.0.1:49808 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 23:22:14.784973 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.782691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.16.62.125:41271)
0926 23:22:14.782875 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.782881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.783389 (+ 508us) server_negotiation.cc:407] Connection header received
0926 23:22:14.783618 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.783625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.783753 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.783858 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.784716 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50448: BlockingRecv error: recv got EOF from 127.0.0.1:50448 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 23:22:14.802193 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.789300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.16.62.125:41271)
0926 23:22:14.789466 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.789471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.800708 (+ 11237us) server_negotiation.cc:407] Connection header received
0926 23:22:14.800886 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.800892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.801000 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.801125 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.801963 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50458: BlockingRecv error: recv got EOF from 127.0.0.1:50458 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 23:22:14.809531 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.807335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.16.62.125:41271)
0926 23:22:14.807503 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.807508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.807983 (+ 475us) server_negotiation.cc:407] Connection header received
0926 23:22:14.808203 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.808208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.808307 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.808418 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.809302 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50460: BlockingRecv error: recv got EOF from 127.0.0.1:50460 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 23:22:14.824076 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.821940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.16.62.125:41271)
0926 23:22:14.822141 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.822147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.822627 (+ 480us) server_negotiation.cc:407] Connection header received
0926 23:22:14.822824 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.822831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.822929 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.823056 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.823832 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50474: BlockingRecv error: recv got EOF from 127.0.0.1:50474 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 23:22:14.838366 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.835253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.16.62.125:41271)
0926 23:22:14.835413 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.835419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.835935 (+ 516us) server_negotiation.cc:407] Connection header received
0926 23:22:14.836117 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.836123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.836255 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.836359 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.838143 (+ 1784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50488: BlockingRecv error: recv got EOF from 127.0.0.1:50488 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 23:22:14.859052 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.856812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.16.62.125:41271)
0926 23:22:14.856979 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.856985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.857502 (+ 517us) server_negotiation.cc:407] Connection header received
0926 23:22:14.857707 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.857713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.857818 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.857929 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.858806 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50500: BlockingRecv error: recv got EOF from 127.0.0.1:50500 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 23:22:14.900463 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.898114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.16.62.125:41271)
0926 23:22:14.898305 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.898312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.898858 (+ 546us) server_negotiation.cc:407] Connection header received
0926 23:22:14.899047 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.899054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.899161 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.899285 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.900180 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50512: BlockingRecv error: recv got EOF from 127.0.0.1:50512 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 23:22:14.974017 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:14.972064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.16.62.125:41271)
0926 23:22:14.972287 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:14.972293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:14.972634 (+ 341us) server_negotiation.cc:407] Connection header received
0926 23:22:14.972821 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:14.972828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:14.972926 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:14.973043 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:14.973784 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50524: BlockingRecv error: recv got EOF from 127.0.0.1:50524 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260926 23:22:15.107780 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:15.105736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.16.62.125:41271)
0926 23:22:15.105949 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:15.105958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:15.106346 (+ 388us) server_negotiation.cc:407] Connection header received
0926 23:22:15.106566 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:15.106573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:15.106699 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:15.106817 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:15.107585 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50528: BlockingRecv error: recv got EOF from 127.0.0.1:50528 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 23:22:15.369122 7069 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:22:15.367383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.16.62.125:41271)
0926 23:22:15.367557 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:15.367561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:15.367881 (+ 320us) server_negotiation.cc:407] Connection header received
0926 23:22:15.368063 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:15.368067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:15.368170 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:15.368264 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:22:15.368956 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50532: BlockingRecv error: recv got EOF from 127.0.0.1:50532 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:22:15.371765 7158 catalog_manager.cc:1259] Loaded cluster ID: 5339ffec62824523a0332f7e916fc2a8
I20260926 23:22:15.371842 7158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c: loading cluster ID for follower catalog manager: success
I20260926 23:22:15.374172 7158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c: acquiring CA information for follower catalog manager: success
I20260926 23:22:15.375794 7158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 691c1195519344729ca2470f8e2f0f0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:22:15.661798 7208 catalog_manager.cc:1259] Loaded cluster ID: 5339ffec62824523a0332f7e916fc2a8
I20260926 23:22:15.661947 7208 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f: loading cluster ID for follower catalog manager: success
I20260926 23:22:15.664835 7208 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f: acquiring CA information for follower catalog manager: success
I20260926 23:22:15.666671 7208 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bfc0f0b5980546bca895ee383bb5d78f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:22:15.892040 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:15.884873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.16.62.125:41271)
0926 23:22:15.885691 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:15.885695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:15.885712 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:22:15.885783 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:15.885786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:15.885927 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:15.886033 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:15.886540 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.887370 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.889354 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.889818 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.889920 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:15.890574 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:15.891165 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:15.891320 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:15.891473 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 23:22:15.891539 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":543,"threads_started":1}
I20260926 23:22:15.905246 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:15.898822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.16.62.124:39623)
0926 23:22:15.899402 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:15.899407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:15.899423 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:22:15.899542 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:15.899547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:15.899684 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:15.899818 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:15.900305 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.901023 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.902947 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.903350 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.903444 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:15.903994 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:15.904587 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:15.904752 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:15.904863 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 23:22:15.904925 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":360,"threads_started":1}
I20260926 23:22:15.926730 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:15.911738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.16.62.125:41271)
0926 23:22:15.911910 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:15.911916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:15.918686 (+ 6770us) server_negotiation.cc:407] Connection header received
0926 23:22:15.919512 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:15.919517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:15.919585 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:15.919635 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:15.920217 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.921705 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.924342 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.925096 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.925186 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:15.925846 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:15.925976 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:15.926043 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:15.926262 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 23:22:15.926369 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 23:22:15.930902 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:15.919245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.16.62.124:39623)
0926 23:22:15.919427 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:15.919434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:15.923581 (+ 4147us) server_negotiation.cc:407] Connection header received
0926 23:22:15.924003 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:15.924009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:15.924151 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:15.924218 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:15.924938 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.926150 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.929211 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.929676 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.929809 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:15.930210 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:15.930310 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:15.930405 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:15.930543 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 23:22:15.930625 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:22:15.936177 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:22:15.911770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.16.62.126:39949)
0926 23:22:15.929978 (+ 18208us) server_negotiation.cc:206] Beginning negotiation
0926 23:22:15.929984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:22:15.930005 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:22:15.930124 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:22:15.930130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:22:15.930370 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:22:15.931219 (+ 849us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:22:15.931392 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.932557 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.933884 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:22:15.934556 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:22:15.934704 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:22:15.935103 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:22:15.935548 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:22:15.935678 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:22:15.935764 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 23:22:15.935847 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18072,"thread_start_us":18775,"threads_started":1}
I20260926 23:22:15.941283 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6924
I20260926 23:22:15.942390 6986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:22:16.052798 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 6924
I20260926 23:22:16.075040 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 6998
I20260926 23:22:16.076179 7060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:22:16.181682 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 6998
I20260926 23:22:16.201647 16633 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu pid 7072
I20260926 23:22:16.202746 7135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:22:16.308646 16633 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZEqmPv/build/asan/bin/kudu with pid 7072
2026-09-26T23:22:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4880 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9822 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (360930 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST